[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:14 dist-test-slave-l5sb krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:14 dist-test-slave-l5sb krb5kdc[399](info): set up 2 sockets
Sep 27 00:44:14 dist-test-slave-l5sb krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:44:15.558317 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:15 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:15.676723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:15.783730 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:15.783960 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:15.783998 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:15.787935 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:15.788051 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:15.788125 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:15.788164 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:15.788201 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:15.793273 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:15.794275 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:15.795259 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:15.803983 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:15.804124 412 server_base.cc:1034] running on GCE node
W20260927 00:44:15.804030 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:15.803996 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:15.804814 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:15.805913 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:15.807075 412 hybrid_clock.cc:648] HybridClock initialized: now 1790469855807031 us; error 54 us; skew 500 ppm
Sep 27 00:44:15 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:15.809749 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:15.810654 412 webserver.cc:460] Webserver started at http://127.0.98.62:45141/ using document root <none> and password file <none>
I20260927 00:44:15.811194 412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:15.811252 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:15.811502 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:15.813459 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "7bf58eb7e7be4d0c804fa2d03a294e44"
format_stamp: "Formatted at 2026-09-27 00:44:15 on dist-test-slave-l5sb"
server_key: "db30d93334c51fb416985f1f8297a804"
server_key_iv: "cbe93b5805b7aa3f156e9d1c6c0f1044"
server_key_version: "encryptionkey@0"
I20260927 00:44:15.814023 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bf58eb7e7be4d0c804fa2d03a294e44"
format_stamp: "Formatted at 2026-09-27 00:44:15 on dist-test-slave-l5sb"
server_key: "db30d93334c51fb416985f1f8297a804"
server_key_iv: "cbe93b5805b7aa3f156e9d1c6c0f1044"
server_key_version: "encryptionkey@0"
I20260927 00:44:15.817550 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:44:15.819800 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:15.820963 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:15.821097 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "7bf58eb7e7be4d0c804fa2d03a294e44"
format_stamp: "Formatted at 2026-09-27 00:44:15 on dist-test-slave-l5sb"
server_key: "db30d93334c51fb416985f1f8297a804"
server_key_iv: "cbe93b5805b7aa3f156e9d1c6c0f1044"
server_key_version: "encryptionkey@0"
I20260927 00:44:15.821204 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:15.848982 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:15.852375 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:15.852598 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:15.859596 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33157
I20260927 00:44:15.859673 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33157 every 8 connection(s)
I20260927 00:44:15.860607 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:15.862669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 412
I20260927 00:44:15.862833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:15.863464 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:15.864107 392 external_mini_cluster.cc:1428] Setting key f11af3191eef359e3cb27535a8bd822e
I20260927 00:44:15.869324 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44: Bootstrap starting.
I20260927 00:44:15.871968 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:15.872710 480 log.cc:824] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:15.874454 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44: No bootstrap required, opened a new log
I20260927 00:44:15.877640 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260927 00:44:15.877919 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:15.878005 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf58eb7e7be4d0c804fa2d03a294e44, State: Initialized, Role: FOLLOWER
I20260927 00:44:15.878445 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260927 00:44:15.878585 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:15.878648 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:15.878741 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:15.879541 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260927 00:44:15.879863 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf58eb7e7be4d0c804fa2d03a294e44; no voters:
I20260927 00:44:15.880142 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:15.880256 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:15.880493 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf58eb7e7be4d0c804fa2d03a294e44, State: Running, Role: LEADER
I20260927 00:44:15.880815 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260927 00:44:15.881345 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 00:44:15 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:15.882254 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf58eb7e7be4d0c804fa2d03a294e44. Latest consensus state: current_term: 1 leader_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } } }
I20260927 00:44:15.882377 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:15.882640 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf58eb7e7be4d0c804fa2d03a294e44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } } }
I20260927 00:44:15.882751 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:15.882988 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:15.886148 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:15.889052 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:15.867130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.98.62:33157)
0927 00:44:15.867575 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:15.867593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:15.868576 (+ 983us) server_negotiation.cc:407] Connection header received
0927 00:44:15.872526 (+ 3950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:15.872541 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:15.872882 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:15.873202 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:15.875384 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:15.876343 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:15.877524 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:15.878136 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:15.878309 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:15.883014 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:15.883020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:15.883037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:15.883071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:15.885075 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:15.885684 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:15.885690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:15.885695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:15.885856 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:15.886558 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:15.886562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:15.886564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:15.886980 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:15.887186 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:15.887772 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 00:44:15.888032 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260927 00:44:15.892204 494 catalog_manager.cc:1347] Generated new cluster ID: 71c65c0f63f243b4a974aa14a90c4aec
I20260927 00:44:15.892264 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:15.903764 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:15.904932 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:15.910682 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf58eb7e7be4d0c804fa2d03a294e44: Generated new TSK 0
I20260927 00:44:15.911456 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:15.941236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:16.051235 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:16.051489 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:16.051533 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:16.055389 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:16.055434 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:16.055490 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:16.060562 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:16.061482 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:16.062439 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:16.071555 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:16.071566 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:16.071642 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:16.071702 506 server_base.cc:1034] running on GCE node
I20260927 00:44:16.072271 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:16.072937 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:16.074117 506 hybrid_clock.cc:648] HybridClock initialized: now 1790469856074106 us; error 32 us; skew 500 ppm
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:16.076823 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:16.077675 506 webserver.cc:460] Webserver started at http://127.0.98.1:40759/ using document root <none> and password file <none>
I20260927 00:44:16.078253 506 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:16.078342 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:16.078575 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:16.080564 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "d50eb5cc4c614b35a9e0b2e91ddf6ce0"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "1a963618b310aa06344c8616c953e006"
server_key_iv: "8f4a913c61647bdb61a6c46311d94e16"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.081173 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d50eb5cc4c614b35a9e0b2e91ddf6ce0"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "1a963618b310aa06344c8616c953e006"
server_key_iv: "8f4a913c61647bdb61a6c46311d94e16"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.084826 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:16.087224 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.088351 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:16.088507 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "d50eb5cc4c614b35a9e0b2e91ddf6ce0"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "1a963618b310aa06344c8616c953e006"
server_key_iv: "8f4a913c61647bdb61a6c46311d94e16"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.088615 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:16.097939 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:16.100522 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:16.100659 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:16.101136 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:16.101940 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:16.101987 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.102025 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:16.102053 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.110644 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42467
I20260927 00:44:16.110666 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42467 every 8 connection(s)
I20260927 00:44:16.111490 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:16.117065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 506
I20260927 00:44:16.117168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:16.117398 392 external_mini_cluster.cc:1428] Setting key 30bc1c32993a802c1e66ac3ce379ca2c
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:16.125375 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.113166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54449 (local address 127.0.98.62:33157)
0927 00:44:16.113292 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:16.113296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:16.114080 (+ 784us) server_negotiation.cc:407] Connection header received
0927 00:44:16.115388 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:16.115391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:16.115448 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:16.115529 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:16.116676 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.117486 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.118761 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.119085 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.119192 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.121479 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:16.121486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:16.121493 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:16.121526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:16.123227 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.123710 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.123718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.123723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.123818 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.124223 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.124229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.124233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.124485 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:16.124601 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:16.125038 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 00:44:16.125182 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:16.126160 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.113477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.1:54449)
0927 00:44:16.113941 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:16.113982 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:16.115189 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:16.115683 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:16.115689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:16.116058 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:16.116532 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.116545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.117609 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.117612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.118621 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.118656 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.119224 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.119227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.119359 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.119961 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:16.119976 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:16.121332 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:16.123344 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.123348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.123358 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.123612 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.124000 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.124003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.124004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.124126 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.124592 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:16.124596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:16.124803 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 00:44:16.124949 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:44:16.125258 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:16.127332 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260927 00:44:16.127573 635 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:16.128058 635 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:16.129549 444 ts_manager.cc:194] Registered new tserver with Master: d50eb5cc4c614b35a9e0b2e91ddf6ce0 (127.0.98.1:42467)
I20260927 00:44:16.130689 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:16.144196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:16.253325 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:16.253599 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:16.253644 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:16.257719 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:16.257808 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:16.257933 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:16.263257 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:16.264262 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:16.265223 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:16.274585 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:16.274730 640 server_base.cc:1034] running on GCE node
W20260927 00:44:16.274585 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:16.274899 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:16.275293 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:16.275877 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:16.277026 640 hybrid_clock.cc:648] HybridClock initialized: now 1790469856277014 us; error 25 us; skew 500 ppm
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:16.279619 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:16.280437 640 webserver.cc:460] Webserver started at http://127.0.98.2:46061/ using document root <none> and password file <none>
I20260927 00:44:16.280915 640 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:16.280972 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:16.281136 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:16.283265 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "27cd8a8136454545969ea586bc1f9a1a"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "e1f3e468534cf426c48468131658447a"
server_key_iv: "74a0166fdfcb9089650b6e4343608511"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.283779 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27cd8a8136454545969ea586bc1f9a1a"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "e1f3e468534cf426c48468131658447a"
server_key_iv: "74a0166fdfcb9089650b6e4343608511"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.287189 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:16.289433 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.290644 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:16.290760 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "27cd8a8136454545969ea586bc1f9a1a"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "e1f3e468534cf426c48468131658447a"
server_key_iv: "74a0166fdfcb9089650b6e4343608511"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.290858 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:16.307523 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:16.310758 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:16.310930 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:16.311537 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:16.312328 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:16.312374 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.312413 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:16.312428 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.321810 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38435
I20260927 00:44:16.321835 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38435 every 8 connection(s)
I20260927 00:44:16.322686 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:16.329843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 640
I20260927 00:44:16.329970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:16.330248 392 external_mini_cluster.cc:1428] Setting key cbd9ce427966de0ceeae42393c726e50
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:16.337054 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.324403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41145 (local address 127.0.98.62:33157)
0927 00:44:16.324562 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:16.324566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:16.325408 (+ 842us) server_negotiation.cc:407] Connection header received
0927 00:44:16.326883 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:16.326886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:16.326982 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:16.327083 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:16.328229 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.328768 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.330124 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.330381 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.330486 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.333292 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:16.333296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:16.333300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:16.333333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:16.334885 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.335506 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.335508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.335510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.335579 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.335963 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.335965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.335966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.336142 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:16.336213 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:16.336829 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:44:16.336925 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:44:16.338363 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.324802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.2:41145)
0927 00:44:16.325246 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:16.325289 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:16.326666 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:16.327254 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:16.327261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:16.327616 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:16.328083 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.328096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.328894 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.328897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.329987 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.329998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.330540 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.330545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.330732 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.331353 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:16.331369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:16.333096 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:16.335004 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.335013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.335030 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.335410 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.335669 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.335675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.335680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.335871 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.336230 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:16.336239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:16.336578 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 00:44:16.336801 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 00:44:16.337185 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260927 00:44:16.339541 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260927 00:44:16.339793 769 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:16.340260 769 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:16.341260 444 ts_manager.cc:194] Registered new tserver with Master: 27cd8a8136454545969ea586bc1f9a1a (127.0.98.2:38435)
I20260927 00:44:16.341881 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:16.357249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:16.464969 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:16.465211 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:16.465293 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:16.469257 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:16.469312 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:16.469425 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:16.474442 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:16.475387 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:16.476332 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:16.485414 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:16.485532 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:16.485427 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:16.485694 774 server_base.cc:1034] running on GCE node
I20260927 00:44:16.486068 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:16.486683 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:16.487864 774 hybrid_clock.cc:648] HybridClock initialized: now 1790469856487845 us; error 28 us; skew 500 ppm
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:16.490639 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:16.491456 774 webserver.cc:460] Webserver started at http://127.0.98.3:33441/ using document root <none> and password file <none>
I20260927 00:44:16.492040 774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:16.492125 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:16.492295 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:16.494333 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d041ff2b17641ee8a6de8b9d7b91df7"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "01f6f24008ddc548ffe54efd9162fe2e"
server_key_iv: "8c7b247b9685b64b8e26a53be9c48a7d"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.494858 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d041ff2b17641ee8a6de8b9d7b91df7"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "01f6f24008ddc548ffe54efd9162fe2e"
server_key_iv: "8c7b247b9685b64b8e26a53be9c48a7d"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.498363 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:16.500535 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.501644 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:16.501760 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "9d041ff2b17641ee8a6de8b9d7b91df7"
format_stamp: "Formatted at 2026-09-27 00:44:16 on dist-test-slave-l5sb"
server_key: "01f6f24008ddc548ffe54efd9162fe2e"
server_key_iv: "8c7b247b9685b64b8e26a53be9c48a7d"
server_key_version: "encryptionkey@0"
I20260927 00:44:16.501885 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:16.513623 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:16.517220 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:16.517416 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:16.518040 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:16.518798 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:16.518842 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.518944 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:16.518980 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:16.527947 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43883
I20260927 00:44:16.528026 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43883 every 8 connection(s)
I20260927 00:44:16.528795 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:16.532730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 774
I20260927 00:44:16.532847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:16.533069 392 external_mini_cluster.cc:1428] Setting key 2bdcd86a22f7ef62d5cf64d7bb48d404
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:16.541878 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.530494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50351 (local address 127.0.98.62:33157)
0927 00:44:16.530621 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:16.530625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:16.531415 (+ 790us) server_negotiation.cc:407] Connection header received
0927 00:44:16.532758 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:16.532761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:16.532815 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:16.532922 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:16.534309 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.535071 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.536332 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.536547 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.536629 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.538959 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:16.538962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:16.538964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:16.538986 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:16.540198 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.540627 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.540630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.540631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.540699 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.540985 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.540987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.540988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.541140 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:16.541210 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:16.541622 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 00:44:16.541709 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:44:16.542637 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.530871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.3:50351)
0927 00:44:16.531273 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:16.531312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:16.532560 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:16.533087 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:16.533094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:16.533416 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:16.534154 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.534175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.535206 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.535209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.536223 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:16.536232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.536669 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.536671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:16.536801 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.537342 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:16.537358 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:16.538811 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:16.540294 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.540299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.540308 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.540547 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.540775 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:16.540777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:16.540778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:16.540901 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:16.541219 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:16.541224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:16.541421 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 00:44:16.541578 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 00:44:16.541863 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260927 00:44:16.543689 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260927 00:44:16.543900 903 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:16.544361 903 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
I20260927 00:44:16.545313 444 ts_manager.cc:194] Registered new tserver with Master: 9d041ff2b17641ee8a6de8b9d7b91df7 (127.0.98.3:43883)
I20260927 00:44:16.545878 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50351
I20260927 00:44:16.547236 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:16.571888 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.563310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.0.98.1:42467)
0927 00:44:16.563613 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:16.563639 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:16.563666 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:44:16.563713 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:16.563718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:16.563913 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:16.564036 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:16.564628 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.565366 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.566216 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.566569 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.566652 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.568889 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:16.568897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:16.568908 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:16.568930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:16.570409 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.570780 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.570784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.570786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.570858 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.571068 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.571070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.571072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.571282 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:16.571424 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:16.571590 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 00:44:16.571740 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":126,"threads_started":1}
W20260927 00:44:16.572805 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33900
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:16.597491 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.589398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.0.98.1:42467)
0927 00:44:16.589542 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:16.589549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:16.589644 (+ 95us) server_negotiation.cc:407] Connection header received
0927 00:44:16.589839 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:16.589843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:16.589895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:16.589992 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:16.590669 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.591188 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.592075 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:16.592304 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:16.592412 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:16.594867 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:16.594870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:16.594872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:16.594893 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:16.596122 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.596473 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.596476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.596477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.596544 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:16.596763 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:16.596765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:16.596766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:16.596910 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:16.597012 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:16.597197 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 00:44:16.597283 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:44:16.598536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 506
I20260927 00:44:16.604152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 640
I20260927 00:44:16.611768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 774
I20260927 00:44:16.616690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 412
2026-09-27T00:44:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2119 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[917](info): set up 2 sockets
Sep 27 00:44:16 dist-test-slave-l5sb krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 00:44:17.673341 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:17 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:17.792220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43753
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:17.899091 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:17.899358 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:17.899420 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:17.903466 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:17.903530 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:17.903549 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:17.903566 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:17.903582 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:17.908789 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:17.909845 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:17.910830 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:17.919116 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:17.919121 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:17.919184 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:17.919389 930 server_base.cc:1034] running on GCE node
I20260927 00:44:17.920027 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:17.921074 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:17.922233 930 hybrid_clock.cc:648] HybridClock initialized: now 1790469857922214 us; error 31 us; skew 500 ppm
Sep 27 00:44:17 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:17.930112 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:17.931022 930 webserver.cc:460] Webserver started at http://127.0.98.62:36765/ using document root <none> and password file <none>
I20260927 00:44:17.931576 930 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:17.931671 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:17.931895 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:17.934058 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "221bb49f047b49b2b061e9e53e90c85e"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-l5sb"
server_key: "1ce485d6da39350779b36871026947b7"
server_key_iv: "247d048aa59c664c37d9a1882484fc20"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.934639 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "221bb49f047b49b2b061e9e53e90c85e"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-l5sb"
server_key: "1ce485d6da39350779b36871026947b7"
server_key_iv: "247d048aa59c664c37d9a1882484fc20"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.938185 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:17.940446 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:17.941577 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:17.941745 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "221bb49f047b49b2b061e9e53e90c85e"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-l5sb"
server_key: "1ce485d6da39350779b36871026947b7"
server_key_iv: "247d048aa59c664c37d9a1882484fc20"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.941931 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:17.951946 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:17.955137 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:17.955328 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:17.962070 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43753
I20260927 00:44:17.962132 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43753 every 8 connection(s)
I20260927 00:44:17.963027 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:17.965730 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:17.967856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 930
I20260927 00:44:17.967985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:17.968217 392 external_mini_cluster.cc:1428] Setting key 36ceaffcf0131f2d5399425b28436d9d
I20260927 00:44:17.971391 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: Bootstrap starting.
I20260927 00:44:17.973465 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:17.974241 998 log.cc:824] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:17.976030 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: No bootstrap required, opened a new log
Sep 27 00:44:17 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:17.978924 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } }
I20260927 00:44:17.979166 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:17.979249 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221bb49f047b49b2b061e9e53e90c85e, State: Initialized, Role: FOLLOWER
I20260927 00:44:17.979715 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } }
I20260927 00:44:17.979873 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:17.979959 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:17.980070 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:17.981132 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } }
I20260927 00:44:17.981472 998 leader_election.cc:302] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221bb49f047b49b2b061e9e53e90c85e; no voters:
I20260927 00:44:17.981813 998 leader_election.cc:288] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:17.981932 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:17.982184 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [term 1 LEADER]: Becoming Leader. State: Replica: 221bb49f047b49b2b061e9e53e90c85e, State: Running, Role: LEADER
I20260927 00:44:17.982497 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } }
I20260927 00:44:17.982632 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:17.969550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.98.62:43753)
0927 00:44:17.970006 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:17.970021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:17.970054 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:44:17.970673 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:17.970689 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:17.971023 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:17.971349 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:17.972102 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:17.973066 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:17.974023 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:17.974535 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:17.974719 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:17.977117 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:17.977122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:17.977134 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:17.977160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:17.979522 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:17.979995 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:17.980002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:17.980008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:17.980152 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:17.980421 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:17.980425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:17.980427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:17.980921 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:17.981133 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:17.981320 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:44:17.981617 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260927 00:44:17.983021 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:17.983616 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221bb49f047b49b2b061e9e53e90c85e. Latest consensus state: current_term: 1 leader_uuid: "221bb49f047b49b2b061e9e53e90c85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } } }
I20260927 00:44:17.983727 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:17.984073 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221bb49f047b49b2b061e9e53e90c85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221bb49f047b49b2b061e9e53e90c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43753 } } }
I20260927 00:44:17.984148 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e [sys.catalog]: This master's current role is: LEADER
W20260927 00:44:17.987535 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:17.987622 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:17.987689 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:17.988334 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:17.993180 1010 catalog_manager.cc:1347] Generated new cluster ID: 4914f9222da64813b460fc72c56c0651
I20260927 00:44:17.993238 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:18.009521 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:18.010766 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:18.029341 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221bb49f047b49b2b061e9e53e90c85e: Generated new TSK 0
I20260927 00:44:18.030078 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:18.058524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.166661 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.166898 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.166940 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.170785 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.170833 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.170889 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:18.175956 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:18.176905 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.177842 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.187462 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.187430 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.187434 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.187820 1024 server_base.cc:1034] running on GCE node
I20260927 00:44:18.188215 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.188858 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.190037 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790469858190023 us; error 25 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.192698 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:18.193589 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40763/ using document root <none> and password file <none>
I20260927 00:44:18.194178 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.194265 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.194473 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.196385 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "caa66711436940f0b3edf384b70ed0fa"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "98e8197b5e8ed735cf3efe7b9fceec4b"
server_key_iv: "e9e6adf47fb4daeb8a8504dce0fe6b02"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.196938 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "caa66711436940f0b3edf384b70ed0fa"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "98e8197b5e8ed735cf3efe7b9fceec4b"
server_key_iv: "e9e6adf47fb4daeb8a8504dce0fe6b02"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.200455 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:44:18.202822 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.204236 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:18.204365 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "caa66711436940f0b3edf384b70ed0fa"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "98e8197b5e8ed735cf3efe7b9fceec4b"
server_key_iv: "e9e6adf47fb4daeb8a8504dce0fe6b02"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.204471 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.224865 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.227797 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.227998 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.228582 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.229367 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.229437 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.229507 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.229560 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.238329 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34843
I20260927 00:44:18.238353 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34843 every 8 connection(s)
I20260927 00:44:18.239193 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:18.244596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1024
I20260927 00:44:18.244704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:18.244980 392 external_mini_cluster.cc:1428] Setting key b2c2335174a4fd1fe514d451b5e4c661
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:18.253916 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.241028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39999 (local address 127.0.98.62:43753)
0927 00:44:18.241175 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.241180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.242070 (+ 890us) server_negotiation.cc:407] Connection header received
0927 00:44:18.243329 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.243331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.243376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.243448 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.244578 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.245346 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.246876 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.247294 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.247414 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.249972 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.249979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.249985 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.250016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.251786 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.252318 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.252321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.252324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.252416 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.252731 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.252738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.252741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.252989 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.253092 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.253551 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.253696 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:18.254743 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.241419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43753 (local address 127.0.98.1:39999)
0927 00:44:18.241927 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.241970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.243145 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.243580 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.243586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.243940 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.244431 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.244444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.245474 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.245479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.246755 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.246765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.247441 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.247444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.247576 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.248197 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.248214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.249725 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.251936 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.251941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.251952 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.252205 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.252501 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.252503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.252504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.252625 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.253087 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.253092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.253308 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.253449 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.253682 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260927 00:44:18.255988 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43753
I20260927 00:44:18.256247 1153 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.256830 1153 heartbeater.cc:511] Master 127.0.98.62:43753 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:18.258543 962 ts_manager.cc:194] Registered new tserver with Master: caa66711436940f0b3edf384b70ed0fa (127.0.98.1:34843)
I20260927 00:44:18.260241 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:18.273160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.382723 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.382978 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.383023 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.387313 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.387384 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.387475 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:18.392740 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:18.393980 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.395032 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.404627 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.404646 1158 server_base.cc:1034] running on GCE node
W20260927 00:44:18.404666 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.404654 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.405508 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.406152 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.407315 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790469858407281 us; error 42 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.410403 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:18.411250 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42677/ using document root <none> and password file <none>
I20260927 00:44:18.411794 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.411883 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.412108 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.414165 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "44c1ebb0e6da4bef8c85c085c6f32545"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "6cca4971a8a791417aaef2a48a34f505"
server_key_iv: "df9bd524fc4dd396ec1d48f5b02b9a37"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.414717 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44c1ebb0e6da4bef8c85c085c6f32545"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "6cca4971a8a791417aaef2a48a34f505"
server_key_iv: "df9bd524fc4dd396ec1d48f5b02b9a37"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.419090 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:44:18.421435 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.422785 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:18.422972 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "44c1ebb0e6da4bef8c85c085c6f32545"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "6cca4971a8a791417aaef2a48a34f505"
server_key_iv: "df9bd524fc4dd396ec1d48f5b02b9a37"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.423082 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.436206 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.439832 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.440052 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.440683 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.441555 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.441604 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.441680 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.441723 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.450985 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40299
I20260927 00:44:18.451007 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40299 every 8 connection(s)
I20260927 00:44:18.452005 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:18.459100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1158
I20260927 00:44:18.459224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:18.459517 392 external_mini_cluster.cc:1428] Setting key 46e0635b828dbb6b5084d88ea01edf2f
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:18.466398 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.453824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49087 (local address 127.0.98.62:43753)
0927 00:44:18.453986 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.453991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.454933 (+ 942us) server_negotiation.cc:407] Connection header received
0927 00:44:18.456197 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.456200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.456249 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.456347 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.457449 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.458122 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.459442 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.459711 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.459814 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.462816 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.462820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.462822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.462846 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.464250 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.464827 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.464831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.464833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.464915 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.465248 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.465253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.465254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.465436 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.465543 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.466106 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.466222 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:44:18.467192 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.454331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43753 (local address 127.0.98.2:49087)
0927 00:44:18.454773 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.454813 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.455987 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.456480 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.456487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.456840 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.457303 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.457316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.458229 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.458232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.459307 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.459317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.459873 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.459878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.460041 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.460707 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.460723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.462619 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.464366 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.464371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.464387 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.464714 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.465014 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.465017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.465019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.465155 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.465539 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.465545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.465855 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.466055 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.466368 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:18.468328 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43753
I20260927 00:44:18.468583 1287 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.469059 1287 heartbeater.cc:511] Master 127.0.98.62:43753 requested a full tablet report, sending...
I20260927 00:44:18.470191 962 ts_manager.cc:194] Registered new tserver with Master: 44c1ebb0e6da4bef8c85c085c6f32545 (127.0.98.2:40299)
I20260927 00:44:18.470849 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:18.486508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.594242 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.594476 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.594523 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.598364 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.598409 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.598464 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:18.603498 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:18.604367 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.605254 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.614313 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.614221 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.614259 1292 server_base.cc:1034] running on GCE node
W20260927 00:44:18.614212 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.614862 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.615487 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.616672 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790469858616657 us; error 26 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.619390 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:18.620256 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43735/ using document root <none> and password file <none>
I20260927 00:44:18.620810 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.620899 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.621122 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.623179 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0284a8c893648f49365381655188d6d"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "83a19e9b85ef4a8d2e5ddd8f6b94a044"
server_key_iv: "b4e55e0c77948ab262fb987daa498c54"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.623723 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0284a8c893648f49365381655188d6d"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "83a19e9b85ef4a8d2e5ddd8f6b94a044"
server_key_iv: "b4e55e0c77948ab262fb987daa498c54"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.627439 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:18.629755 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.630972 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:18.631104 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "d0284a8c893648f49365381655188d6d"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-l5sb"
server_key: "83a19e9b85ef4a8d2e5ddd8f6b94a044"
server_key_iv: "b4e55e0c77948ab262fb987daa498c54"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.631208 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.663232 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.666680 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.666886 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.667507 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.668318 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.668388 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.668460 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.668510 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.677039 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45131
I20260927 00:44:18.677076 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45131 every 8 connection(s)
I20260927 00:44:18.677918 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:18.682376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1292
I20260927 00:44:18.682511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:18.682832 392 external_mini_cluster.cc:1428] Setting key a98bb4b1afc560a70477f7a541be8a6e
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:18.691226 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.679650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58473 (local address 127.0.98.62:43753)
0927 00:44:18.679780 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.679784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.680603 (+ 819us) server_negotiation.cc:407] Connection header received
0927 00:44:18.681884 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.681887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.681932 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.682007 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.683324 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.684054 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.685368 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.685598 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.685704 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.688033 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.688037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.688042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.688072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.689248 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.689733 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.689737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.689739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.689866 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.690199 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.690202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.690204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.690379 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.690482 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.690945 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.691064 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:18.691982 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.680030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43753 (local address 127.0.98.3:58473)
0927 00:44:18.680443 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.680495 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.681651 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.682166 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.682173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.682579 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.683181 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.683197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.684200 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.684206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.685234 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.685243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.685737 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.685740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.685973 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.686490 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.686505 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.687868 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.689359 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.689365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.689381 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.689631 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.689961 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.689964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.689966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.690098 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.690485 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.690494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.690722 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.690906 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.691199 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260927 00:44:18.693018 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43753
I20260927 00:44:18.693230 1421 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.693645 1421 heartbeater.cc:511] Master 127.0.98.62:43753 requested a full tablet report, sending...
I20260927 00:44:18.694669 962 ts_manager.cc:194] Registered new tserver with Master: d0284a8c893648f49365381655188d6d (127.0.98.3:45131)
I20260927 00:44:18.695322 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58473
I20260927 00:44:18.696489 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:18.707332 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.700458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.98.62:43753)
0927 00:44:18.700561 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.700565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.700776 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:44:18.700867 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.700869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.700902 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.700965 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.701594 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.702148 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.702919 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.703155 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.703232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.704441 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.704444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.704446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.704467 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.705959 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.706336 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.706339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.706340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.706410 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.706635 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.706637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.706638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.706832 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.706933 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.707097 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.707186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:44:18.712112 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:18.715129 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:18.729199 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.723986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.98.2:40299)
0927 00:44:18.724660 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.724671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.724691 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:44:18.724760 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.724765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.724961 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.725107 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.725250 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726361 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.727934 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.728465 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.728772 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.728918 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.728968 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.729066 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":88,"threads_started":1}
I20260927 00:44:18.729195 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.722907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34843 (local address 127.0.0.1:37748)
0927 00:44:18.723650 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.723695 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.723825 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.724575 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.724581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.724601 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.724727 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.724733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726164 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.726170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.727570 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.727579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.728557 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.728563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.728704 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.728768 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.728870 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.729040 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":147,"thread_start_us":68,"threads_started":1}
I20260927 00:44:18.729233 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.723582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40299 (local address 127.0.0.1:46500)
0927 00:44:18.724336 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.724349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.724441 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.725079 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.725084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.725109 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.725164 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.725167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726483 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.726489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.727796 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.727807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.728612 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.728615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.728716 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.728808 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.728870 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.729056 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":113,"spinlock_wait_cycles":8960,"thread_start_us":97,"threads_started":1}
I20260927 00:44:18.729560 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.723529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.98.3:45131)
0927 00:44:18.724159 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.724172 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.724404 (+ 232us) server_negotiation.cc:407] Connection header received
0927 00:44:18.724547 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.724552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.724725 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.724863 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.725254 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726046 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.727974 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.728721 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.728856 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.728991 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.729299 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.729391 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":150,"threads_started":1}
I20260927 00:44:18.729964 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.722980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.0.1:36126)
0927 00:44:18.724314 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.724328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.724440 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.724863 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.724866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.724880 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.724988 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.724994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726664 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.726667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.727866 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.727877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.729050 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.729053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.729147 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.729168 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.729732 (+ 564us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.729844 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":47,"spinlock_wait_cycles":4224,"thread_start_us":107,"threads_started":1}
I20260927 00:44:18.730134 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.723183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.98.1:34843)
0927 00:44:18.723500 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.723518 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.724025 (+ 507us) server_negotiation.cc:407] Connection header received
0927 00:44:18.724111 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.724116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.724436 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.724586 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.724842 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.726031 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.727692 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.728420 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.728533 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.728672 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.729880 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.729979 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260927 00:44:18.732574 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 9e99ea530659413bb02b50ac36505aef (DEFAULT_TABLE table=test-table [id=8a2410207b004168895a7c002755fb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.732504 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 9e99ea530659413bb02b50ac36505aef (DEFAULT_TABLE table=test-table [id=8a2410207b004168895a7c002755fb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.732831 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 9e99ea530659413bb02b50ac36505aef (DEFAULT_TABLE table=test-table [id=8a2410207b004168895a7c002755fb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.733582 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e99ea530659413bb02b50ac36505aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.733582 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e99ea530659413bb02b50ac36505aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.733750 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e99ea530659413bb02b50ac36505aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.739751 1438 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Bootstrap starting.
I20260927 00:44:18.741729 1439 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Bootstrap starting.
I20260927 00:44:18.742089 1440 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: Bootstrap starting.
I20260927 00:44:18.742195 1438 tablet_bootstrap.cc:654] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.743248 1438 log.cc:824] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.743875 1440 tablet_bootstrap.cc:654] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.744349 1439 tablet_bootstrap.cc:654] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.745023 1438 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: No bootstrap required, opened a new log
I20260927 00:44:18.745167 1439 log.cc:824] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.745191 1440 log.cc:824] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.745221 1438 ts_tablet_manager.cc:1397] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:18.747092 1439 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: No bootstrap required, opened a new log
I20260927 00:44:18.747277 1439 ts_tablet_manager.cc:1397] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:18.747398 1440 tablet_bootstrap.cc:492] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: No bootstrap required, opened a new log
I20260927 00:44:18.747567 1440 ts_tablet_manager.cc:1397] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:18.749498 1438 raft_consensus.cc:348] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.749786 1438 raft_consensus.cc:374] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.749871 1438 raft_consensus.cc:729] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0284a8c893648f49365381655188d6d, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.750245 1440 raft_consensus.cc:348] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.750330 1438 consensus_queue.cc:260] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.750440 1440 raft_consensus.cc:374] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.750507 1440 raft_consensus.cc:729] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa66711436940f0b3edf384b70ed0fa, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.750910 1440 consensus_queue.cc:260] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.751273 1438 ts_tablet_manager.cc:1428] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 00:44:18.751258 1439 raft_consensus.cc:348] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.751514 1439 raft_consensus.cc:374] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.751575 1439 raft_consensus.cc:729] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1ebb0e6da4bef8c85c085c6f32545, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.751729 1421 heartbeater.cc:503] Master 127.0.98.62:43753 was elected leader, sending a full tablet report...
I20260927 00:44:18.751796 1153 heartbeater.cc:503] Master 127.0.98.62:43753 was elected leader, sending a full tablet report...
I20260927 00:44:18.751958 1439 consensus_queue.cc:260] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.752046 1440 ts_tablet_manager.cc:1428] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260927 00:44:18.752805 1439 ts_tablet_manager.cc:1428] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:44:18.753110 1287 heartbeater.cc:503] Master 127.0.98.62:43753 was elected leader, sending a full tablet report...
I20260927 00:44:18.833351 1446 raft_consensus.cc:482] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:18.833570 1446 raft_consensus.cc:504] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.834872 1446 leader_election.cc:288] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa66711436940f0b3edf384b70ed0fa (127.0.98.1:34843), d0284a8c893648f49365381655188d6d (127.0.98.3:45131)
I20260927 00:44:18.838454 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.835086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.98.2:46657)
0927 00:44:18.835207 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.835224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.835346 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.835616 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.835620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.835637 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.835768 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.835774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.836520 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.836525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.837339 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.837348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.838140 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.838142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.838221 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.838240 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.838299 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.838350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 00:44:18.838527 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.835238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46657 (local address 127.0.98.3:45131)
0927 00:44:18.835379 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.835383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.835396 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:18.835442 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.835444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.835498 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.835603 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.835900 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.836404 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.837479 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.838026 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.838135 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.838199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.838337 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.838404 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:44:18.839177 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.835086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34843 (local address 127.0.98.2:55085)
0927 00:44:18.835763 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.835780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.835882 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.836128 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.836131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.836145 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.836275 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.836280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.836982 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.836987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.837749 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.837757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.838780 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.838783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.838882 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.838914 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.838990 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.839052 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":332,"thread_start_us":134,"threads_started":1}
I20260927 00:44:18.839186 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.835238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55085 (local address 127.0.98.1:34843)
0927 00:44:18.835384 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.835390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.835852 (+ 462us) server_negotiation.cc:407] Connection header received
0927 00:44:18.835972 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.835975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.836033 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.836117 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.836392 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.836872 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.837916 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.838539 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.838651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.838722 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.839026 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.839098 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:44:18.839157 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e99ea530659413bb02b50ac36505aef" candidate_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0284a8c893648f49365381655188d6d" is_pre_election: true
I20260927 00:44:18.839478 1375 raft_consensus.cc:2454] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1ebb0e6da4bef8c85c085c6f32545 in term 0.
I20260927 00:44:18.839779 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e99ea530659413bb02b50ac36505aef" candidate_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa66711436940f0b3edf384b70ed0fa" is_pre_election: true
I20260927 00:44:18.840082 1107 raft_consensus.cc:2454] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1ebb0e6da4bef8c85c085c6f32545 in term 0.
I20260927 00:44:18.840101 1174 leader_election.cc:302] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1ebb0e6da4bef8c85c085c6f32545, d0284a8c893648f49365381655188d6d; no voters:
I20260927 00:44:18.840377 1446 raft_consensus.cc:2781] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:18.840473 1446 raft_consensus.cc:482] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:18.840520 1446 raft_consensus.cc:3037] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.841555 1446 raft_consensus.cc:504] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.841923 1446 leader_election.cc:288] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [CANDIDATE]: Term 1 election: Requested vote from peers caa66711436940f0b3edf384b70ed0fa (127.0.98.1:34843), d0284a8c893648f49365381655188d6d (127.0.98.3:45131)
I20260927 00:44:18.842273 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e99ea530659413bb02b50ac36505aef" candidate_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa66711436940f0b3edf384b70ed0fa"
I20260927 00:44:18.842407 1107 raft_consensus.cc:3037] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.842473 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e99ea530659413bb02b50ac36505aef" candidate_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0284a8c893648f49365381655188d6d"
I20260927 00:44:18.842662 1375 raft_consensus.cc:3037] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.843480 1107 raft_consensus.cc:2454] T 9e99ea530659413bb02b50ac36505aef P caa66711436940f0b3edf384b70ed0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1ebb0e6da4bef8c85c085c6f32545 in term 1.
I20260927 00:44:18.843838 1375 raft_consensus.cc:2454] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1ebb0e6da4bef8c85c085c6f32545 in term 1.
I20260927 00:44:18.843847 1177 leader_election.cc:302] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1ebb0e6da4bef8c85c085c6f32545, caa66711436940f0b3edf384b70ed0fa; no voters:
I20260927 00:44:18.844026 1446 raft_consensus.cc:2781] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:18.844249 1446 raft_consensus.cc:686] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [term 1 LEADER]: Becoming Leader. State: Replica: 44c1ebb0e6da4bef8c85c085c6f32545, State: Running, Role: LEADER
I20260927 00:44:18.844625 1446 consensus_queue.cc:237] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } }
I20260927 00:44:18.848088 961 catalog_manager.cc:5495] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c1ebb0e6da4bef8c85c085c6f32545 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa66711436940f0b3edf384b70ed0fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0284a8c893648f49365381655188d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c1ebb0e6da4bef8c85c085c6f32545" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40299 } health_report { overall_health: HEALTHY } } }
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:18.890268 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.881173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.98.1:34843)
0927 00:44:18.881350 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.881355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.881500 (+ 145us) server_negotiation.cc:407] Connection header received
0927 00:44:18.881663 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.881667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.881721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.881850 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.882573 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.883106 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.884023 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.884241 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.884352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.886959 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.886972 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.886987 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.887011 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.888637 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.889101 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.889107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.889109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.889202 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.889440 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.889442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.889443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.889674 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.889900 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.890049 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.890146 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:18.918074 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.909452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.98.1:34843)
0927 00:44:18.909627 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.909632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.909821 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:44:18.910054 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.910058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.910113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.910201 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.910892 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.911419 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.912304 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.912551 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.912659 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.915113 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.915116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.915119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.915141 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.916410 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.916826 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.916831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.916833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.916908 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.917181 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.917185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.917186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.917348 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.917483 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.917760 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.917900 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 00:44:18.928791 1422 tablet.cc:2369] T 9e99ea530659413bb02b50ac36505aef P d0284a8c893648f49365381655188d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:44:18 dist-test-slave-l5sb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:18.944339 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.936258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.0.98.1:34843)
0927 00:44:18.936435 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.936439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.936544 (+ 105us) server_negotiation.cc:407] Connection header received
0927 00:44:18.936616 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.936619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.936662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.936735 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.937347 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.937986 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.938869 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.939075 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.939160 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.941350 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.941353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.941356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.941378 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.942847 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.943191 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.943194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.943195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.943263 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.943564 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.943567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.943569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.943729 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.943852 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.944015 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.944104 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:44:18.945742 1067 tablet_service.cc:2908] Rejecting scan request for tablet 9e99ea530659413bb02b50ac36505aef: Uninitialized: clean time has not yet been initialized
I20260927 00:44:18.947751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1024
W20260927 00:44:18.953235 1288 tablet.cc:2369] T 9e99ea530659413bb02b50ac36505aef P 44c1ebb0e6da4bef8c85c085c6f32545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:18.953562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1158
I20260927 00:44:18.959450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1292
I20260927 00:44:18.965942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 930
2026-09-27T00:44:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2349 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:19 dist-test-slave-l5sb krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:19 dist-test-slave-l5sb krb5kdc[1462](info): set up 2 sockets
Sep 27 00:44:19 dist-test-slave-l5sb krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 00:44:20.045408 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:20.160511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41203
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:20.267877 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:20.268131 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:20.268172 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:20.272094 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:20.272140 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:44:20.272156 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:20.272173 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:20.272238 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:20.272256 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:20.277339 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:20.278347 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:20.279332 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:20.287470 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:20.287463 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:20.287474 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:20.287886 1475 server_base.cc:1034] running on GCE node
I20260927 00:44:20.288467 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:20.289546 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:20.290757 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790469860290723 us; error 55 us; skew 500 ppm
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:20.293395 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:20.294581 1475 webserver.cc:460] Webserver started at http://127.0.98.62:43455/ using document root <none> and password file <none>
I20260927 00:44:20.295207 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:20.295296 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:20.295506 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:20.297621 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "6704b9130a144c46849c570891a71a00"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "d9cf98e9ffbb5d0b118f31278b53fc5c"
server_key_iv: "85c6aaa5fcb78794772f08d312b856cf"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.298225 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "6704b9130a144c46849c570891a71a00"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "d9cf98e9ffbb5d0b118f31278b53fc5c"
server_key_iv: "85c6aaa5fcb78794772f08d312b856cf"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.301858 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:44:20.304260 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:20.305447 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:20.305603 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "6704b9130a144c46849c570891a71a00"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "d9cf98e9ffbb5d0b118f31278b53fc5c"
server_key_iv: "85c6aaa5fcb78794772f08d312b856cf"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.305711 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:20.331151 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:20.333987 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:20.334193 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:20.341990 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41203
I20260927 00:44:20.342036 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41203 every 8 connection(s)
I20260927 00:44:20.342901 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:20.345541 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:20.346304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1475
I20260927 00:44:20.346423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:20.346691 392 external_mini_cluster.cc:1428] Setting key f3e5b2c3d59177213ba51b0da179d676
I20260927 00:44:20.351647 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Bootstrap starting.
I20260927 00:44:20.353857 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:20.354570 1553 log.cc:824] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:20.356663 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: No bootstrap required, opened a new log
I20260927 00:44:20.360178 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } }
I20260927 00:44:20.360412 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:20.360488 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6704b9130a144c46849c570891a71a00, State: Initialized, Role: FOLLOWER
I20260927 00:44:20.360940 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } }
I20260927 00:44:20.361088 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:20.361171 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:20.361241 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.348039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.98.62:41203)
0927 00:44:20.348482 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:20.348495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:20.348533 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:44:20.349153 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:20.349169 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:20.349480 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:20.349838 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:20.350600 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.351553 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.352433 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.352898 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.353078 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.355808 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:20.355814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:20.355827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:20.355851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:20.358174 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.358684 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.358689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.358693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.358814 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.359099 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.359101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.359102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.359453 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:20.359625 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:20.359906 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:44:20.360309 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260927 00:44:20.361285 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:20.362224 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } }
I20260927 00:44:20.362545 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6704b9130a144c46849c570891a71a00; no voters:
I20260927 00:44:20.362864 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:20.362933 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:20.363139 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [term 1 LEADER]: Becoming Leader. State: Replica: 6704b9130a144c46849c570891a71a00, State: Running, Role: LEADER
I20260927 00:44:20.363476 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } }
I20260927 00:44:20.364080 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:20.365082 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6704b9130a144c46849c570891a71a00. Latest consensus state: current_term: 1 leader_uuid: "6704b9130a144c46849c570891a71a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } } }
I20260927 00:44:20.365206 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:20.365476 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6704b9130a144c46849c570891a71a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6704b9130a144c46849c570891a71a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41203 } } }
I20260927 00:44:20.365563 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:20.365576 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:20.368178 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:20.373415 1567 catalog_manager.cc:1347] Generated new cluster ID: a928072305984a23b2d80f5d2c716fb6
I20260927 00:44:20.373474 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:20.397006 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:20.398313 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:20.413950 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Generated new TSK 0
I20260927 00:44:20.414702 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:20.446440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41203
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:20.553663 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:20.553973 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:20.554018 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:20.557945 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:20.557997 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:20.558065 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:20.558142 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:20.563334 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:20.564278 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:20.565280 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:20.574234 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:20.574474 1579 server_base.cc:1034] running on GCE node
W20260927 00:44:20.574229 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:20.574226 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:20.574951 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:20.575515 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:20.576705 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790469860576696 us; error 29 us; skew 500 ppm
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:20.579290 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:20.580085 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45843/ using document root <none> and password file <none>
I20260927 00:44:20.580619 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:20.580708 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:20.580924 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:20.582967 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "b52335c9fdfc40509ee1c972b4bac6f3"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "78045ca1bd81462c0358dc6ca2dc776f"
server_key_iv: "3bb157ee1684df3098c16dca81112652"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.583525 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b52335c9fdfc40509ee1c972b4bac6f3"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "78045ca1bd81462c0358dc6ca2dc776f"
server_key_iv: "3bb157ee1684df3098c16dca81112652"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.587097 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:20.590134 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:20.591288 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:20.591408 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "b52335c9fdfc40509ee1c972b4bac6f3"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "78045ca1bd81462c0358dc6ca2dc776f"
server_key_iv: "3bb157ee1684df3098c16dca81112652"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.591522 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:20.606611 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:20.609338 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:20.609572 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:20.611058 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:20.611133 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:20.611200 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:20.611248 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:20.621753 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43259
I20260927 00:44:20.621827 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43259 every 8 connection(s)
I20260927 00:44:20.622634 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:20.625429 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.611870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38127 (local address 127.0.98.62:41203)
0927 00:44:20.612005 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:20.612008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:20.613021 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 00:44:20.614144 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:20.614150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:20.614215 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:20.614280 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:20.616041 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.616929 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.618446 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.618703 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.618821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.621733 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:20.621738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:20.621740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:20.621793 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:20.623444 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.623932 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.623937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.623938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.624026 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.624365 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.624369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.624371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.624555 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:20.624658 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:20.625134 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 00:44:20.625258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:20.626212 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.612414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.1:38127)
0927 00:44:20.612863 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:20.612902 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:20.613890 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:20.614744 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:20.614752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:20.615242 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:20.615883 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.615907 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.617056 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.617061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.618326 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.618334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.618859 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.618863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.619045 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.619668 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:20.619697 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:20.621551 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:20.623561 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.623566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.623580 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.623828 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.624125 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.624128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.624130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.624271 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.624666 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:20.624675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:20.624906 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 00:44:20.625083 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:44:20.625384 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260927 00:44:20.627473 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41203
I20260927 00:44:20.627601 1710 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:20.628144 1710 heartbeater.cc:511] Master 127.0.98.62:41203 requested a full tablet report, sending...
I20260927 00:44:20.629701 1506 ts_manager.cc:194] Registered new tserver with Master: b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1:43259)
I20260927 00:44:20.630856 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38127
I20260927 00:44:20.632462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1579
I20260927 00:44:20.632591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:20.632861 392 external_mini_cluster.cc:1428] Setting key 522e768b97ab6c062972f64688f65d45
I20260927 00:44:20.637382 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.629716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39759 (local address 127.0.98.62:41203)
0927 00:44:20.629873 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:20.629879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:20.630080 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:44:20.630216 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:20.630219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:20.630260 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:20.630344 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:20.630763 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.631335 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.632249 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.632470 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.632576 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.633759 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:20.633761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:20.633763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:20.633824 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:20.635440 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.635832 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.635838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.635843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.635934 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.636262 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.636268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.636272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.636646 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:20.636824 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:20.637135 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 00:44:20.637274 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:44:20.637392 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.629637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.1:39759)
0927 00:44:20.630000 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:20.630012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:20.630119 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:20.630370 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:20.630373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:20.630567 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:20.630655 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.630660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.631484 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.631488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.632128 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.632137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.632581 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.632585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.632684 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.633188 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:20.633211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:20.633647 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:20.635557 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.635559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.635561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.635740 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.636051 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.636056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.636060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.636172 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.636784 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:20.636789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:20.636897 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:44:20.637008 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:44:20.637181 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:20.660166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41203
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:20.771488 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:20.771833 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:20.771924 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:20.778383 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:20.778468 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:20.778525 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:20.778666 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:20.784420 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:20.785429 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:20.786479 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:20.795375 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:20.795411 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:20.795372 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:20.795670 1719 server_base.cc:1034] running on GCE node
I20260927 00:44:20.796165 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:20.796790 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:20.797971 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790469860797954 us; error 26 us; skew 500 ppm
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:20.800634 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:20.801467 1719 webserver.cc:460] Webserver started at http://127.0.98.2:38253/ using document root <none> and password file <none>
I20260927 00:44:20.802052 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:20.802137 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:20.802351 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:20.804267 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "88b53d07a3324d6897734b970aef5334"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "01985b37d253d41f13ad870dd0f96740"
server_key_iv: "2ea1ff37a5cb98b041d4f8a75db9dc8e"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.804800 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "88b53d07a3324d6897734b970aef5334"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "01985b37d253d41f13ad870dd0f96740"
server_key_iv: "2ea1ff37a5cb98b041d4f8a75db9dc8e"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.808257 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:20.810674 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:20.811822 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:20.811949 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "88b53d07a3324d6897734b970aef5334"
format_stamp: "Formatted at 2026-09-27 00:44:20 on dist-test-slave-l5sb"
server_key: "01985b37d253d41f13ad870dd0f96740"
server_key_iv: "2ea1ff37a5cb98b041d4f8a75db9dc8e"
server_key_version: "encryptionkey@0"
I20260927 00:44:20.812055 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:20.828405 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:20.831595 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:20.831785 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:20.833169 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:20.833238 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:20.833308 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:20.833364 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:20 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:20.844295 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43621
I20260927 00:44:20.844332 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43621 every 8 connection(s)
I20260927 00:44:20.845304 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:20.845968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1719
I20260927 00:44:20.846112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:20.846410 392 external_mini_cluster.cc:1428] Setting key 2bb2711df879fe353987ad27fad34d6a
I20260927 00:44:20.846848 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.833886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51937 (local address 127.0.98.62:41203)
0927 00:44:20.834003 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:20.834006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:20.834998 (+ 992us) server_negotiation.cc:407] Connection header received
0927 00:44:20.836116 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:20.836120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:20.836182 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:20.836242 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:20.837533 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.838476 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.839713 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.840013 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.840122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.843024 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:20.843026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:20.843028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:20.843050 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:20.844629 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.845168 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.845171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.845172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.845243 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.845591 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.845594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.845595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.845846 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:20.845945 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:20.846580 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 00:44:20.846687 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:20.847707 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.834413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.2:51937)
0927 00:44:20.834859 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:20.834898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:20.835845 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:20.836628 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:20.836643 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:20.836941 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:20.837385 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.837397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.838608 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.838611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.839589 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.839598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.840171 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.840175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.840344 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.840914 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:20.840942 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:20.842861 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:20.844733 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.844738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.844751 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.845072 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.845333 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.845339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.845342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.845494 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.846004 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:20.846013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:20.846313 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 00:44:20.846533 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 00:44:20.846851 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260927 00:44:20.848915 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41203
I20260927 00:44:20.849047 1850 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:20.849557 1850 heartbeater.cc:511] Master 127.0.98.62:41203 requested a full tablet report, sending...
I20260927 00:44:20.850602 1506 ts_manager.cc:194] Registered new tserver with Master: 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621)
I20260927 00:44:20.851337 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51937
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:44:20.860002 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.851157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55857 (local address 127.0.98.62:41203)
0927 00:44:20.851276 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:20.851280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:20.852781 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 00:44:20.852864 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:20.852867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:20.852910 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:20.852987 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:20.853378 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.854297 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.855197 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.855561 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.855675 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.856881 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:20.856887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:20.856890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:20.856917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:20.858284 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.858692 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.858696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.858697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.858813 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:20.859061 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:20.859063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:20.859065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:20.859352 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:20.859459 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:20.859721 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 00:44:20.859836 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:20.860018 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:20.851081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.2:55857)
0927 00:44:20.852688 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:20.852700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:20.852780 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:20.853023 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:20.853025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:20.853180 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:20.853268 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.853273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.854441 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.854444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.855073 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:20.855082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:20.855683 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:20.855686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:20.855761 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:20.856362 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:20.856376 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:20.856717 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:20.858384 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.858387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.858389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.858574 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.858909 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:20.858911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:20.858912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:20.858984 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:20.859449 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:20.859453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:20.859560 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:44:20.859669 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:44:20.859815 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:20.876029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41203
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:20.984633 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:20.984884 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:20.984925 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:20.988955 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:20.989006 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:20.989079 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:20.989163 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:20.994208 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:20.995234 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:20.996268 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:21.005359 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:21.005364 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:21.005595 1859 server_base.cc:1034] running on GCE node
W20260927 00:44:21.005364 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:21.006016 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:21.006631 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:21.007809 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790469861007797 us; error 23 us; skew 500 ppm
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:21.010656 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:21.011475 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38675/ using document root <none> and password file <none>
I20260927 00:44:21.011956 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:21.012019 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:21.012180 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:21.014108 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "21ae7467fc45436b937e06d6bf46fe5d"
format_stamp: "Formatted at 2026-09-27 00:44:21 on dist-test-slave-l5sb"
server_key: "7f058bd9c0babfd1b73500aee70efd2e"
server_key_iv: "82544f6e2a75e7467033069d81e8a44b"
server_key_version: "encryptionkey@0"
I20260927 00:44:21.014645 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21ae7467fc45436b937e06d6bf46fe5d"
format_stamp: "Formatted at 2026-09-27 00:44:21 on dist-test-slave-l5sb"
server_key: "7f058bd9c0babfd1b73500aee70efd2e"
server_key_iv: "82544f6e2a75e7467033069d81e8a44b"
server_key_version: "encryptionkey@0"
I20260927 00:44:21.018114 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:21.020315 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:21.021489 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:21.021577 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "21ae7467fc45436b937e06d6bf46fe5d"
format_stamp: "Formatted at 2026-09-27 00:44:21 on dist-test-slave-l5sb"
server_key: "7f058bd9c0babfd1b73500aee70efd2e"
server_key_iv: "82544f6e2a75e7467033069d81e8a44b"
server_key_version: "encryptionkey@0"
I20260927 00:44:21.021653 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:21.043432 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:21.046491 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:21.046644 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:21.047983 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:21.048039 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:21.048105 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:21.048135 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:21.058712 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39059
I20260927 00:44:21.059087 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39059 every 8 connection(s)
I20260927 00:44:21.059631 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:21.061758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 1859
I20260927 00:44:21.061941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:21.061990 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.048933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53577 (local address 127.0.98.62:41203)
0927 00:44:21.049076 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.049079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.050220 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 00:44:21.051295 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.051299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.051383 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.051483 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.053035 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.053970 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.055278 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.055519 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.055632 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.058330 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.058335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.058339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.058375 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.060150 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.060658 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.060661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.060662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.060740 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.060973 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.060975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.060976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.061134 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.061203 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.061685 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.061821 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:44:21.062281 392 external_mini_cluster.cc:1428] Setting key 552fa1f3ea9095fb9d1f2a84cd24d704
I20260927 00:44:21.062904 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.049515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.3:53577)
0927 00:44:21.050021 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.050108 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.051068 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.051714 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.051723 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.052157 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:21.052873 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.052891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.054102 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.054106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.055152 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.055162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.055664 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.055668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.055852 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.056419 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:21.056434 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:21.058160 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:21.060275 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.060280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.060294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.060581 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.060806 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.060807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.060809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.060920 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.061217 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:21.061221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:21.061426 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.061634 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.061949 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260927 00:44:21.064117 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41203
I20260927 00:44:21.064262 1990 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:21.064944 1990 heartbeater.cc:511] Master 127.0.98.62:41203 requested a full tablet report, sending...
I20260927 00:44:21.066056 1506 ts_manager.cc:194] Registered new tserver with Master: 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059)
I20260927 00:44:21.066666 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53577
I20260927 00:44:21.072597 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.066164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40301 (local address 127.0.98.62:41203)
0927 00:44:21.066279 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.066284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.066450 (+ 166us) server_negotiation.cc:407] Connection header received
0927 00:44:21.066564 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.066567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.066605 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.066694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.067206 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.067936 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.068705 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.068904 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.068982 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.069980 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.069983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.069985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.070011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.071402 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.071735 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.071737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.071738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.071805 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.072015 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.072017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.072018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.072161 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.072230 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.072405 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.072483 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:21.072599 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.066121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.98.3:40301)
0927 00:44:21.066374 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.066385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.066489 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.066707 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.066709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.066941 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:21.067076 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.067086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.068068 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.068071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.068606 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.068614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.068997 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.068999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.069072 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.069488 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:21.069498 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:21.069862 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:21.071488 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.071490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.071492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.071658 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.071878 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.071879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.071881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.071945 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.072234 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:21.072236 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:21.072300 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.072374 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.072466 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260927 00:44:21.076128 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:21.110194 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.101163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.62:41203)
0927 00:44:21.101335 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.101340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.101355 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:21.101425 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.101429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.101504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.101596 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.102648 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.103284 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.104687 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.104902 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.104979 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.107394 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.107398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.107399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.107422 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.108700 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.109116 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.109119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.109120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.109190 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.109425 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.109427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.109428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.109583 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.109681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.109910 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.110000 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:21.113009 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:21.116243 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:21.130187 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.123954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43259 (local address 127.0.0.1:40568)
0927 00:44:21.124817 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.124866 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.125063 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.125790 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.125796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.125819 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.126031 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.126039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.127509 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.127512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.128654 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.128664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.129624 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.129627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.129730 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.129762 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.129927 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.130040 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":49,"spinlock_wait_cycles":2176,"thread_start_us":72,"threads_started":1}
I20260927 00:44:21.130707 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.124086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.0.98.1:43259)
0927 00:44:21.124533 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.124549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.124950 (+ 401us) server_negotiation.cc:407] Connection header received
0927 00:44:21.125238 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.125243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.125448 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.125577 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.126155 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.127364 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.128825 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.129495 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.129629 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.129768 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.130476 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.130581 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":111,"threads_started":1}
I20260927 00:44:21.131196 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.125111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.2:43621)
0927 00:44:21.125997 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.126005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.126023 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:44:21.126081 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.126085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.126285 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.126437 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.126642 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.127709 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.129212 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.130042 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.130157 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.130314 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.130965 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.131067 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":111,"threads_started":1}
I20260927 00:44:21.131402 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.124175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.0.1:49906)
0927 00:44:21.124825 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.124865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.125063 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.125835 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.125838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.125850 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.126031 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.126039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.128773 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.128777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.129808 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.129819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.130981 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.130985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.131089 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.131116 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.131214 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.131277 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":24,"spinlock_wait_cycles":6528,"thread_start_us":87,"threads_started":1}
I20260927 00:44:21.131454 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.124351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.0.98.3:39059)
0927 00:44:21.124716 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.124726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.124949 (+ 223us) server_negotiation.cc:407] Connection header received
0927 00:44:21.125237 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.125243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.125448 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.125578 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.127512 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.128663 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.129930 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.130670 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.130776 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.130923 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.131243 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.131324 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":85,"threads_started":1}
I20260927 00:44:21.131419 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.124644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43621 (local address 127.0.0.1:50584)
0927 00:44:21.125340 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.125353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.125439 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.126414 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.126416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.126431 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.126506 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.126511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.127866 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.127869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.129092 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.129101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.130155 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.130159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.130497 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.130522 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.130599 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.130676 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260927 00:44:21.133519 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 (DEFAULT_TABLE table=test-table [id=ae79ed9484fb4439bd0148400f1329f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.134521 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c273210e24c53b9c74b1d67b447b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.134330 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 (DEFAULT_TABLE table=test-table [id=ae79ed9484fb4439bd0148400f1329f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.134330 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 (DEFAULT_TABLE table=test-table [id=ae79ed9484fb4439bd0148400f1329f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.135309 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c273210e24c53b9c74b1d67b447b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.135300 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c273210e24c53b9c74b1d67b447b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.140347 2013 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Bootstrap starting.
I20260927 00:44:21.142875 2013 tablet_bootstrap.cc:654] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.143019 2014 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Bootstrap starting.
I20260927 00:44:21.144207 2013 log.cc:824] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:21.144852 2015 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Bootstrap starting.
I20260927 00:44:21.145300 2014 tablet_bootstrap.cc:654] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.146077 2013 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: No bootstrap required, opened a new log
I20260927 00:44:21.146180 2014 log.cc:824] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:21.146260 2013 ts_tablet_manager.cc:1397] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:44:21.146893 2015 tablet_bootstrap.cc:654] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.147478 2015 log.cc:824] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:21.148100 2014 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: No bootstrap required, opened a new log
I20260927 00:44:21.148288 2014 ts_tablet_manager.cc:1397] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:21.149164 2015 tablet_bootstrap.cc:492] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: No bootstrap required, opened a new log
I20260927 00:44:21.149377 2015 ts_tablet_manager.cc:1397] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:44:21.150200 2013 raft_consensus.cc:348] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.150465 2013 raft_consensus.cc:374] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.150565 2013 raft_consensus.cc:729] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52335c9fdfc40509ee1c972b4bac6f3, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.151024 2013 consensus_queue.cc:260] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.151638 1710 heartbeater.cc:503] Master 127.0.98.62:41203 was elected leader, sending a full tablet report...
I20260927 00:44:21.151999 2013 ts_tablet_manager.cc:1428] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:44:21.152791 2015 raft_consensus.cc:348] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.153059 2015 raft_consensus.cc:374] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.152930 2014 raft_consensus.cc:348] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.153142 2015 raft_consensus.cc:729] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ae7467fc45436b937e06d6bf46fe5d, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.153191 2014 raft_consensus.cc:374] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.153285 2014 raft_consensus.cc:729] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b53d07a3324d6897734b970aef5334, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.153551 2015 consensus_queue.cc:260] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.153759 2014 consensus_queue.cc:260] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.154227 1990 heartbeater.cc:503] Master 127.0.98.62:41203 was elected leader, sending a full tablet report...
I20260927 00:44:21.154732 2014 ts_tablet_manager.cc:1428] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 00:44:21.155066 1850 heartbeater.cc:503] Master 127.0.98.62:41203 was elected leader, sending a full tablet report...
I20260927 00:44:21.155584 2015 ts_tablet_manager.cc:1428] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 00:44:21.255659 2019 raft_consensus.cc:482] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:21.255890 2019 raft_consensus.cc:504] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.257047 2019 leader_election.cc:288] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621), 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059)
I20260927 00:44:21.260900 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.257292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.98.1:50379)
0927 00:44:21.257701 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.257716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.257876 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.258171 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.258173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.258188 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.258275 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.258279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.259012 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.259016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.259852 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.259860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.260474 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.260479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.260568 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.260597 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.260671 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.260721 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":120,"threads_started":1}
I20260927 00:44:21.260905 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.257436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50379 (local address 127.0.98.3:39059)
0927 00:44:21.257630 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.257637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.257819 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:44:21.257979 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.257981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.258056 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.258129 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.258407 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.258892 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.259974 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.260352 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.260430 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.260517 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.260706 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.260785 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 00:44:21.260995 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.257411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59415 (local address 127.0.98.2:43621)
0927 00:44:21.257612 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.257617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.257701 (+ 84us) server_negotiation.cc:407] Connection header received
0927 00:44:21.257893 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.257898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.257961 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.258036 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.258377 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.258895 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.260014 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.260383 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.260513 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.260576 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.260795 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.260860 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 00:44:21.260965 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.257277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43621 (local address 127.0.98.1:59415)
0927 00:44:21.257599 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.257614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.257735 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.258082 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.258088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.258109 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.258255 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.258267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.259006 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.259010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.259860 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.259869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.260527 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.260531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.260640 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.260684 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.260761 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.260818 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":50,"thread_start_us":124,"threads_started":1}
I20260927 00:44:21.261686 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c273210e24c53b9c74b1d67b447b1" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ae7467fc45436b937e06d6bf46fe5d" is_pre_election: true
I20260927 00:44:21.261686 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c273210e24c53b9c74b1d67b447b1" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334" is_pre_election: true
I20260927 00:44:21.261982 1804 raft_consensus.cc:2454] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 0.
I20260927 00:44:21.262005 1944 raft_consensus.cc:2454] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 0.
I20260927 00:44:21.262475 1596 leader_election.cc:302] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21ae7467fc45436b937e06d6bf46fe5d, 88b53d07a3324d6897734b970aef5334, b52335c9fdfc40509ee1c972b4bac6f3; no voters:
I20260927 00:44:21.262727 2019 raft_consensus.cc:2781] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:21.262807 2019 raft_consensus.cc:482] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:21.262848 2019 raft_consensus.cc:3037] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.263931 2019 raft_consensus.cc:504] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.264357 2019 leader_election.cc:288] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621), 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059)
I20260927 00:44:21.264627 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c273210e24c53b9c74b1d67b447b1" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334"
I20260927 00:44:21.264679 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c273210e24c53b9c74b1d67b447b1" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ae7467fc45436b937e06d6bf46fe5d"
I20260927 00:44:21.264775 1804 raft_consensus.cc:3037] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.264776 1944 raft_consensus.cc:3037] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.266166 1804 raft_consensus.cc:2454] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 1.
I20260927 00:44:21.266434 1944 raft_consensus.cc:2454] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 1.
I20260927 00:44:21.266466 1596 leader_election.cc:302] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b53d07a3324d6897734b970aef5334, b52335c9fdfc40509ee1c972b4bac6f3; no voters:
I20260927 00:44:21.266635 2019 raft_consensus.cc:2781] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:21.266884 2019 raft_consensus.cc:686] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 LEADER]: Becoming Leader. State: Replica: b52335c9fdfc40509ee1c972b4bac6f3, State: Running, Role: LEADER
I20260927 00:44:21.267237 2019 consensus_queue.cc:237] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.270273 1505 catalog_manager.cc:5495] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:21.290788 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.287433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.1:43259)
0927 00:44:21.287556 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.287560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.287573 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:21.287758 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.287760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.287813 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.287894 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:21.288137 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.288608 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.289453 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.289651 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.289722 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.290052 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.290418 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.290500 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.290622 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.290686 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:44:21.296692 1944 raft_consensus.cc:1261] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Refusing update from remote peer b52335c9fdfc40509ee1c972b4bac6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.296983 1804 raft_consensus.cc:1261] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Refusing update from remote peer b52335c9fdfc40509ee1c972b4bac6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.297413 2024 consensus_queue.cc:1035] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.297639 2019 consensus_queue.cc:1035] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.316301 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:21.316476 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:21.318804 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Sending DeleteTablet for 3 replicas of tablet 0a5c273210e24c53b9c74b1d67b447b1
I20260927 00:44:21.319545 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49906
I20260927 00:44:21.319567 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50584
I20260927 00:44:21.320179 2038 tablet_replica.cc:330] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: stopping tablet replica
I20260927 00:44:21.320139 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c273210e24c53b9c74b1d67b447b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40568
I20260927 00:44:21.320241 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:44:21.320540 2038 raft_consensus.cc:2229] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:21.320550 2037 tablet_replica.cc:330] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: stopping tablet replica
W20260927 00:44:21.320618 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:21.320818 2037 raft_consensus.cc:2229] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:21.320840 2038 raft_consensus.cc:2258] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.321090 2037 raft_consensus.cc:2258] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.321398 2039 tablet_replica.cc:330] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: stopping tablet replica
I20260927 00:44:21.321643 2039 raft_consensus.cc:2229] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:21.322062 2039 raft_consensus.cc:2258] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.322309 2038 ts_tablet_manager.cc:1905] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.323624 2037 ts_tablet_manager.cc:1905] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.325353 2038 ts_tablet_manager.cc:1918] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:21.325511 2038 log.cc:1196] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/0a5c273210e24c53b9c74b1d67b447b1
I20260927 00:44:21.325943 2038 ts_tablet_manager.cc:1939] T 0a5c273210e24c53b9c74b1d67b447b1 P 21ae7467fc45436b937e06d6bf46fe5d: Deleting consensus metadata
I20260927 00:44:21.326579 2037 ts_tablet_manager.cc:1918] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:21.326737 2037 log.cc:1196] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/0a5c273210e24c53b9c74b1d67b447b1
I20260927 00:44:21.326967 1644 tablet_service.cc:1467] Processing CreateTablet for tablet d316ab6eab564af39b6855ee8ccfda8e (DEFAULT_TABLE table=test-table [id=97f7def0821e4a1482b4b0c5461eb985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.327142 2037 ts_tablet_manager.cc:1939] T 0a5c273210e24c53b9c74b1d67b447b1 P 88b53d07a3324d6897734b970aef5334: Deleting consensus metadata
I20260927 00:44:21.327226 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d316ab6eab564af39b6855ee8ccfda8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.328116 1492 catalog_manager.cc:4841] TS 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621): tablet 0a5c273210e24c53b9c74b1d67b447b1 (table test-table [id=ae79ed9484fb4439bd0148400f1329f6]) successfully deleted
I20260927 00:44:21.329496 2013 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Bootstrap starting.
I20260927 00:44:21.329648 1784 tablet_service.cc:1467] Processing CreateTablet for tablet d316ab6eab564af39b6855ee8ccfda8e (DEFAULT_TABLE table=test-table [id=97f7def0821e4a1482b4b0c5461eb985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.329896 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d316ab6eab564af39b6855ee8ccfda8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.330145 1924 tablet_service.cc:1467] Processing CreateTablet for tablet d316ab6eab564af39b6855ee8ccfda8e (DEFAULT_TABLE table=test-table [id=97f7def0821e4a1482b4b0c5461eb985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:21.330372 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d316ab6eab564af39b6855ee8ccfda8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.330463 2013 tablet_bootstrap.cc:654] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.331908 2014 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Bootstrap starting.
I20260927 00:44:21.332424 1491 catalog_manager.cc:4841] TS 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059): tablet 0a5c273210e24c53b9c74b1d67b447b1 (table test-table [id=ae79ed9484fb4439bd0148400f1329f6]) successfully deleted
I20260927 00:44:21.332811 2014 tablet_bootstrap.cc:654] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.332998 2013 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: No bootstrap required, opened a new log
I20260927 00:44:21.333077 2013 ts_tablet_manager.cc:1397] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:44:21.333340 2015 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Bootstrap starting.
I20260927 00:44:21.333367 2039 ts_tablet_manager.cc:1905] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.333492 2013 raft_consensus.cc:348] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.333591 2013 raft_consensus.cc:374] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.333621 2013 raft_consensus.cc:729] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52335c9fdfc40509ee1c972b4bac6f3, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.333738 2013 consensus_queue.cc:260] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.334000 2013 ts_tablet_manager.cc:1428] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:21.334280 2015 tablet_bootstrap.cc:654] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.335843 2039 ts_tablet_manager.cc:1918] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:21.335997 2039 log.cc:1196] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/0a5c273210e24c53b9c74b1d67b447b1
I20260927 00:44:21.336372 2039 ts_tablet_manager.cc:1939] T 0a5c273210e24c53b9c74b1d67b447b1 P b52335c9fdfc40509ee1c972b4bac6f3: Deleting consensus metadata
I20260927 00:44:21.337090 2015 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: No bootstrap required, opened a new log
I20260927 00:44:21.337173 2015 ts_tablet_manager.cc:1397] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 00:44:21.337491 1492 catalog_manager.cc:4841] TS b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1:43259): tablet 0a5c273210e24c53b9c74b1d67b447b1 (table test-table [id=ae79ed9484fb4439bd0148400f1329f6]) successfully deleted
I20260927 00:44:21.337658 2015 raft_consensus.cc:348] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.337805 2015 raft_consensus.cc:374] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.337852 2015 raft_consensus.cc:729] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ae7467fc45436b937e06d6bf46fe5d, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.338008 2015 consensus_queue.cc:260] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.338270 2015 ts_tablet_manager.cc:1428] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:21.339026 2014 tablet_bootstrap.cc:492] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: No bootstrap required, opened a new log
I20260927 00:44:21.339110 2014 ts_tablet_manager.cc:1397] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260927 00:44:21.339545 2014 raft_consensus.cc:348] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.339646 2014 raft_consensus.cc:374] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.339668 2014 raft_consensus.cc:729] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b53d07a3324d6897734b970aef5334, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.339820 2014 consensus_queue.cc:260] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.340094 2014 ts_tablet_manager.cc:1428] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 00:44:21.347518 1851 tablet.cc:2369] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:21.373950 1711 tablet.cc:2369] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:21.500310 2033 raft_consensus.cc:482] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:21.500458 2033 raft_consensus.cc:504] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.500851 2033 leader_election.cc:288] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621), 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059)
I20260927 00:44:21.501335 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d316ab6eab564af39b6855ee8ccfda8e" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334" is_pre_election: true
I20260927 00:44:21.501322 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d316ab6eab564af39b6855ee8ccfda8e" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ae7467fc45436b937e06d6bf46fe5d" is_pre_election: true
I20260927 00:44:21.501493 1804 raft_consensus.cc:2454] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 0.
I20260927 00:44:21.501493 1944 raft_consensus.cc:2454] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 0.
I20260927 00:44:21.501817 1595 leader_election.cc:302] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ae7467fc45436b937e06d6bf46fe5d, b52335c9fdfc40509ee1c972b4bac6f3; no voters:
I20260927 00:44:21.502012 2033 raft_consensus.cc:2781] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:21.502107 2033 raft_consensus.cc:482] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:21.502162 2033 raft_consensus.cc:3037] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.503125 2033 raft_consensus.cc:504] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.503459 2033 leader_election.cc:288] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621), 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059)
I20260927 00:44:21.503834 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d316ab6eab564af39b6855ee8ccfda8e" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ae7467fc45436b937e06d6bf46fe5d"
I20260927 00:44:21.503937 1944 raft_consensus.cc:3037] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.503896 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d316ab6eab564af39b6855ee8ccfda8e" candidate_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334"
I20260927 00:44:21.504047 1804 raft_consensus.cc:3037] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.504889 1944 raft_consensus.cc:2454] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 1.
I20260927 00:44:21.504908 1804 raft_consensus.cc:2454] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52335c9fdfc40509ee1c972b4bac6f3 in term 1.
I20260927 00:44:21.505190 1595 leader_election.cc:302] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21ae7467fc45436b937e06d6bf46fe5d, 88b53d07a3324d6897734b970aef5334, b52335c9fdfc40509ee1c972b4bac6f3; no voters:
I20260927 00:44:21.505363 2033 raft_consensus.cc:2781] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:21.505484 2033 raft_consensus.cc:686] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 LEADER]: Becoming Leader. State: Replica: b52335c9fdfc40509ee1c972b4bac6f3, State: Running, Role: LEADER
I20260927 00:44:21.505626 2033 consensus_queue.cc:237] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.507166 1505 catalog_manager.cc:5495] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:21.559221 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.551596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.98.62:41203)
0927 00:44:21.551736 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.551743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.551938 (+ 195us) server_negotiation.cc:407] Connection header received
0927 00:44:21.552089 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.552092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.552134 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.552218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.552861 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.553370 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.554198 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.554402 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.554502 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.556596 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.556599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.556601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.556621 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.557810 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.558219 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.558221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.558222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.558290 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.558526 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.558528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.558529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.558676 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.558746 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.559010 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.559102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:44:21.559223 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.551564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41203 (local address 127.0.0.1:40658)
0927 00:44:21.551847 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.551858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.551987 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.552234 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.552236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.552657 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:21.552744 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.552750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.553472 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.553478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.554090 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.554099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.554514 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.554524 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.554612 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.555104 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:21.555114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:21.556462 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:21.557908 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.557915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.557920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.558138 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.558365 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:21.558367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:21.558368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:21.558446 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:21.558760 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:21.558765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:21.558870 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.558958 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.559058 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260927 00:44:21.561286 1991 tablet.cc:2369] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:21.562870 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40658:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:44:21.563572 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:21.568044 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e5529a68da27455e8743929f2747c83b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=047828cfb26343d0bededdc3d5b17f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:21.568169 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e5529a68da27455e8743929f2747c83b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=047828cfb26343d0bededdc3d5b17f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:21.568223 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e5529a68da27455e8743929f2747c83b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=047828cfb26343d0bededdc3d5b17f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:21.568426 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5529a68da27455e8743929f2747c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.568521 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5529a68da27455e8743929f2747c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.568540 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5529a68da27455e8743929f2747c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:21.571054 2014 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: Bootstrap starting.
I20260927 00:44:21.571071 2013 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: Bootstrap starting.
I20260927 00:44:21.571808 2014 tablet_bootstrap.cc:654] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.571810 2013 tablet_bootstrap.cc:654] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.572396 2015 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: Bootstrap starting.
I20260927 00:44:21.573278 2014 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: No bootstrap required, opened a new log
I20260927 00:44:21.573318 2015 tablet_bootstrap.cc:654] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:21.573334 2014 ts_tablet_manager.cc:1397] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 00:44:21.573444 2013 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: No bootstrap required, opened a new log
I20260927 00:44:21.573521 2013 ts_tablet_manager.cc:1397] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:21.573655 2014 raft_consensus.cc:348] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.573733 2014 raft_consensus.cc:374] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.573753 2014 raft_consensus.cc:729] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b53d07a3324d6897734b970aef5334, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.573926 2014 consensus_queue.cc:260] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.573958 2013 raft_consensus.cc:348] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.574101 2013 raft_consensus.cc:374] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.574139 2013 raft_consensus.cc:729] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52335c9fdfc40509ee1c972b4bac6f3, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.574200 2014 ts_tablet_manager.cc:1428] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:21.574260 2013 consensus_queue.cc:260] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.574514 2013 ts_tablet_manager.cc:1428] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:21.574528 2015 tablet_bootstrap.cc:492] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: No bootstrap required, opened a new log
I20260927 00:44:21.574595 2015 ts_tablet_manager.cc:1397] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:44:21.574752 2021 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.574874 2021 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.575027 2015 raft_consensus.cc:348] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.575052 2033 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.575140 2015 raft_consensus.cc:374] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:21.575160 2033 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.575174 2015 raft_consensus.cc:729] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ae7467fc45436b937e06d6bf46fe5d, State: Initialized, Role: FOLLOWER
I20260927 00:44:21.575294 2015 consensus_queue.cc:260] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.575570 2015 ts_tablet_manager.cc:1428] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:21.575533 2020 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.575665 2020 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.613739 2020 raft_consensus.cc:482] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:21.613922 2020 raft_consensus.cc:504] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.614893 2020 leader_election.cc:288] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1:43259), 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621)
I20260927 00:44:21.619398 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.615604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35103 (local address 127.0.98.2:43621)
0927 00:44:21.615716 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.615721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.615893 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:44:21.616075 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.616079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.616123 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.616184 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.616629 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.617290 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618399 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.618790 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618908 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.618978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.619189 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.619266 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:44:21.619418 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.615229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56919 (local address 127.0.98.1:43259)
0927 00:44:21.615403 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.615410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.615483 (+ 73us) server_negotiation.cc:407] Connection header received
0927 00:44:21.615658 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.615661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.615718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.615797 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:21.616588 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.617253 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618388 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.618795 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.618977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.619211 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.619291 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:44:21.619444 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.615509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43621 (local address 127.0.98.3:35103)
0927 00:44:21.615821 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.615833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.615945 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.616415 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.616418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.616430 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.616539 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.616547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.617414 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.617417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.618280 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.618289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.618907 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.619035 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.619087 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.619174 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.619274 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":38,"thread_start_us":80,"threads_started":1}
I20260927 00:44:21.619453 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.615141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43259 (local address 127.0.98.3:56919)
0927 00:44:21.615381 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.615394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.615534 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.615829 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.615833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.615857 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:21.616429 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.616436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.617365 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.617370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.618267 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.618275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.618912 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.618915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.619091 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.619108 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.619193 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.619291 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":69,"thread_start_us":70,"threads_started":1}
I20260927 00:44:21.620066 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5529a68da27455e8743929f2747c83b" candidate_uuid: "21ae7467fc45436b937e06d6bf46fe5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334" is_pre_election: true
I20260927 00:44:21.620064 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5529a68da27455e8743929f2747c83b" candidate_uuid: "21ae7467fc45436b937e06d6bf46fe5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" is_pre_election: true
I20260927 00:44:21.620191 1804 raft_consensus.cc:2454] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21ae7467fc45436b937e06d6bf46fe5d in term 0.
I20260927 00:44:21.620306 1664 raft_consensus.cc:2454] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21ae7467fc45436b937e06d6bf46fe5d in term 0.
I20260927 00:44:21.620569 1876 leader_election.cc:302] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ae7467fc45436b937e06d6bf46fe5d, 88b53d07a3324d6897734b970aef5334; no voters:
I20260927 00:44:21.620836 2020 raft_consensus.cc:2781] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:21.620885 2020 raft_consensus.cc:482] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:21.620911 2020 raft_consensus.cc:3037] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.622062 2020 raft_consensus.cc:504] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.622377 2020 leader_election.cc:288] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [CANDIDATE]: Term 1 election: Requested vote from peers b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1:43259), 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621)
I20260927 00:44:21.622761 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5529a68da27455e8743929f2747c83b" candidate_uuid: "21ae7467fc45436b937e06d6bf46fe5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52335c9fdfc40509ee1c972b4bac6f3"
I20260927 00:44:21.622882 1664 raft_consensus.cc:3037] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.623759 1664 raft_consensus.cc:2454] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21ae7467fc45436b937e06d6bf46fe5d in term 1.
I20260927 00:44:21.624048 1876 leader_election.cc:302] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ae7467fc45436b937e06d6bf46fe5d, b52335c9fdfc40509ee1c972b4bac6f3; no voters:
I20260927 00:44:21.624254 2020 raft_consensus.cc:2781] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:21.624451 2020 raft_consensus.cc:686] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [term 1 LEADER]: Becoming Leader. State: Replica: 21ae7467fc45436b937e06d6bf46fe5d, State: Running, Role: LEADER
I20260927 00:44:21.624715 2020 consensus_queue.cc:237] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260927 00:44:21.625926 2055 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: TxnStatusTablet state changed. Reason: New leader 21ae7467fc45436b937e06d6bf46fe5d. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.626085 2055 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:21.626360 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:44:21.626502 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:44:21.626900 1506 catalog_manager.cc:5495] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: HEALTHY } } }
I20260927 00:44:21.627887 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5529a68da27455e8743929f2747c83b" candidate_uuid: "21ae7467fc45436b937e06d6bf46fe5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b53d07a3324d6897734b970aef5334"
I20260927 00:44:21.628023 1804 raft_consensus.cc:3037] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:21.629204 1804 raft_consensus.cc:2454] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21ae7467fc45436b937e06d6bf46fe5d in term 1.
I20260927 00:44:21.654675 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.650758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.0.98.3:39059)
0927 00:44:21.650890 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.650893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.650925 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:44:21.651120 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.651123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.651173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.651254 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:21.651535 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.652059 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.653290 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.653502 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.653589 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.653939 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.654328 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.654417 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.654512 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.654557 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:21.654731 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.650639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.0.1:49922)
0927 00:44:21.650870 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.650884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.650984 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.651288 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.651291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.651300 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:21.651427 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.651433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.652163 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.652167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.653138 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.653149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.653618 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.653621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.653712 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.653760 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.654422 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.654435 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.654506 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.654595 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 00:44:21.658000 1804 raft_consensus.cc:1261] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Refusing update from remote peer 21ae7467fc45436b937e06d6bf46fe5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.658093 1664 raft_consensus.cc:1261] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Refusing update from remote peer 21ae7467fc45436b937e06d6bf46fe5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.658540 2020 consensus_queue.cc:1035] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.658700 2055 consensus_queue.cc:1035] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.660091 2021 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: TxnStatusTablet state changed. Reason: New leader 21ae7467fc45436b937e06d6bf46fe5d. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.660208 2021 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.660805 2033 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: TxnStatusTablet state changed. Reason: New leader 21ae7467fc45436b937e06d6bf46fe5d. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.660919 2033 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.661638 2055 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.661734 2055 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:21.663702 2033 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.663805 2033 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P b52335c9fdfc40509ee1c972b4bac6f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.663897 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333764553345691648 to 7333764553217314816. Current Snapshot: MvccSnapshot[applied={T|T < 7333764553345691648 or (T in {7333764553345691648})}]
I20260927 00:44:21.665388 2020 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.665484 2020 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 21ae7467fc45436b937e06d6bf46fe5d: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:21.666535 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333764553345691648 to 7333764553217314816. Current Snapshot: MvccSnapshot[applied={T|T < 7333764553345691648 or (T in {7333764553345691648})}]
I20260927 00:44:21.667827 2021 tablet_replica.cc:439] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21ae7467fc45436b937e06d6bf46fe5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52335c9fdfc40509ee1c972b4bac6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43259 } } peers { permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 } } peers { permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } }
I20260927 00:44:21.667943 2021 tablet_replica.cc:442] T e5529a68da27455e8743929f2747c83b P 88b53d07a3324d6897734b970aef5334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:21.676168 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.672701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59147 (local address 127.0.98.3:39059)
0927 00:44:21.672821 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.672826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.673010 (+ 184us) server_negotiation.cc:407] Connection header received
0927 00:44:21.673122 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.673124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.673162 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.673222 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:21.673447 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.674032 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.675209 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.675408 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.675481 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.675776 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.675832 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.675882 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.676041 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.676077 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:44:21.676450 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.672634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.98.1:59147)
0927 00:44:21.672920 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.672935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.673037 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.673249 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.673252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.673260 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:21.673362 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.673368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.674128 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.674131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.675101 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.675111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.675505 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.675509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.675607 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.675655 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.675915 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.675922 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.676033 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.676133 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260927 00:44:21.682832 1804 raft_consensus.cc:1261] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Refusing update from remote peer b52335c9fdfc40509ee1c972b4bac6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.682837 1944 raft_consensus.cc:1261] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Refusing update from remote peer b52335c9fdfc40509ee1c972b4bac6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:21.683238 2019 consensus_queue.cc:1035] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ae7467fc45436b937e06d6bf46fe5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.683404 2033 consensus_queue.cc:1035] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b53d07a3324d6897734b970aef5334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:21.701454 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333764553449803776 to 7333764552728317952. Current Snapshot: MvccSnapshot[applied={T|T < 7333764553449803776}]
I20260927 00:44:21.703761 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.699184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35069 (local address 127.0.98.1:43259)
0927 00:44:21.699277 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.699281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.699608 (+ 327us) server_negotiation.cc:407] Connection header received
0927 00:44:21.699813 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.699817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.699871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.699934 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:21.700390 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.701160 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.702351 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.702665 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.702760 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.703174 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.703250 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.703298 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.703582 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.703638 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 00:44:21.703794 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.698981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43259 (local address 127.0.98.3:35069)
0927 00:44:21.699529 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:21.699546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:21.699681 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:21.700074 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:21.700078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:21.700088 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:21.700264 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.700273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.701294 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.701296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.702227 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:21.702238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.702866 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.702870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:21.702978 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.703031 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:21.703472 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:21.703477 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:44:21.703544 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:44:21.703617 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":111,"threads_started":1}
I20260927 00:44:21.728698 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:21.728854 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:21.730795 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6704b9130a144c46849c570891a71a00: Sending DeleteTablet for 3 replicas of tablet d316ab6eab564af39b6855ee8ccfda8e
I20260927 00:44:21.731381 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet d316ab6eab564af39b6855ee8ccfda8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49906
I20260927 00:44:21.731400 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet d316ab6eab564af39b6855ee8ccfda8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40568
I20260927 00:44:21.731557 2038 tablet_replica.cc:330] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: stopping tablet replica
I20260927 00:44:21.731575 2039 tablet_replica.cc:330] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: stopping tablet replica
I20260927 00:44:21.731662 2038 raft_consensus.cc:2229] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:21.731670 2039 raft_consensus.cc:2229] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:21.731758 2038 raft_consensus.cc:2258] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.731830 2039 raft_consensus.cc:2258] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.732620 2038 ts_tablet_manager.cc:1905] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.732616 2039 ts_tablet_manager.cc:1905] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.734776 2038 ts_tablet_manager.cc:1918] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:21.734867 2038 log.cc:1196] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/d316ab6eab564af39b6855ee8ccfda8e
I20260927 00:44:21.735095 2038 ts_tablet_manager.cc:1939] T d316ab6eab564af39b6855ee8ccfda8e P 21ae7467fc45436b937e06d6bf46fe5d: Deleting consensus metadata
I20260927 00:44:21.735249 2039 ts_tablet_manager.cc:1918] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:21.735328 2039 log.cc:1196] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/d316ab6eab564af39b6855ee8ccfda8e
I20260927 00:44:21.735589 2039 ts_tablet_manager.cc:1939] T d316ab6eab564af39b6855ee8ccfda8e P b52335c9fdfc40509ee1c972b4bac6f3: Deleting consensus metadata
I20260927 00:44:21.735756 1491 catalog_manager.cc:4841] TS 21ae7467fc45436b937e06d6bf46fe5d (127.0.98.3:39059): tablet d316ab6eab564af39b6855ee8ccfda8e (table test-table [id=97f7def0821e4a1482b4b0c5461eb985]) successfully deleted
I20260927 00:44:21.735839 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet d316ab6eab564af39b6855ee8ccfda8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50584
I20260927 00:44:21.735998 2037 tablet_replica.cc:330] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: stopping tablet replica
I20260927 00:44:21.736151 2037 raft_consensus.cc:2229] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:21.736224 1492 catalog_manager.cc:4841] TS b52335c9fdfc40509ee1c972b4bac6f3 (127.0.98.1:43259): tablet d316ab6eab564af39b6855ee8ccfda8e (table test-table [id=97f7def0821e4a1482b4b0c5461eb985]) successfully deleted
I20260927 00:44:21.736272 2037 raft_consensus.cc:2258] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:21.736935 2037 ts_tablet_manager.cc:1905] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:21.739012 2037 ts_tablet_manager.cc:1918] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:21.739102 2037 log.cc:1196] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/d316ab6eab564af39b6855ee8ccfda8e
I20260927 00:44:21.739378 2037 ts_tablet_manager.cc:1939] T d316ab6eab564af39b6855ee8ccfda8e P 88b53d07a3324d6897734b970aef5334: Deleting consensus metadata
I20260927 00:44:21.740388 1492 catalog_manager.cc:4841] TS 88b53d07a3324d6897734b970aef5334 (127.0.98.2:43621): tablet d316ab6eab564af39b6855ee8ccfda8e (table test-table [id=97f7def0821e4a1482b4b0c5461eb985]) successfully deleted
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:21.769186 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.736716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.0.98.1:43259)
0927 00:44:21.736815 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.736819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.737050 (+ 231us) server_negotiation.cc:407] Connection header received
0927 00:44:21.737218 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.737221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.737268 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.737316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.738286 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.739032 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.740197 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.740563 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.740671 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.764559 (+ 23888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.764577 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.764595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.764641 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.767066 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.767560 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.767568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.767570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.767662 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.767945 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.767951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.767955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.768301 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.768511 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.768900 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.769027 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:44:21.769793 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40596
I20260927 00:44:21.780953 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:21.771612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.0.98.62:41203)
0927 00:44:21.771743 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:21.771747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:21.771998 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:44:21.772103 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:21.772106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:21.772167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:21.772252 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:21.773249 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.774077 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.775413 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:21.775851 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:21.775948 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:21.777443 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:21.777447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:21.777450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:21.777487 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:21.779236 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.779708 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.779711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.779713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.779815 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:21.780096 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:21.780101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:21.780102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:21.780280 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:21.780417 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:21.780639 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:44:21.780754 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:44:21.781348 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40668
I20260927 00:44:21.782787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1579
I20260927 00:44:21.791443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1719
I20260927 00:44:21.798821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1859
I20260927 00:44:21.805401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 1475
2026-09-27T00:44:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2839 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[2077](info): set up 2 sockets
Sep 27 00:44:21 dist-test-slave-l5sb krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260927 00:44:22.939041 392 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 00:44:23.139889 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.123s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/test-user.keytab.
W20260927 00:44:23.736626 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.538s user 0.000s sys 0.002s
Sep 27 00:44:23 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 00:44:24.145323 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.387s user 0.000s sys 0.003s
I20260927 00:44:24.145903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:24.255664 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:24.256009 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:24.256084 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:24.260265 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:24.260351 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:44:24.260385 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:24.260449 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:24.260506 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:24.260543 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:24.265910 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:24.266985 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:24.268023 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:24.276669 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.276795 2090 server_base.cc:1034] running on GCE node
W20260927 00:44:24.276576 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:24.276576 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.277539 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:24.278734 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:24.279908 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790469864279888 us; error 33 us; skew 500 ppm
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:24.288388 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:24.289391 2090 webserver.cc:460] Webserver started at http://127.0.98.62:44335/ using document root <none> and password file <none>
I20260927 00:44:24.290014 2090 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:24.290082 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:24.290293 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:24.292424 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "36278c05cd6d4e159370c7bffa47069b"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "5ed5cea8d046fb3f979b7b2565b2c7a2"
server_key_iv: "e2fefe485c8af82ad81001ca3cdb9a05"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.292932 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "36278c05cd6d4e159370c7bffa47069b"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "5ed5cea8d046fb3f979b7b2565b2c7a2"
server_key_iv: "e2fefe485c8af82ad81001ca3cdb9a05"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.296674 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:44:24.299141 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:24.300429 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:24.300562 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "36278c05cd6d4e159370c7bffa47069b"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "5ed5cea8d046fb3f979b7b2565b2c7a2"
server_key_iv: "e2fefe485c8af82ad81001ca3cdb9a05"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.300678 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:24.317862 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:24.321497 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:24.321707 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:24.330157 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46821
I20260927 00:44:24.330176 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46821 every 8 connection(s)
I20260927 00:44:24.331156 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:24.332330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2090
I20260927 00:44:24.332461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:24.332729 392 external_mini_cluster.cc:1428] Setting key 74ffe482fa6cd115bdb1510f4f98ed88
I20260927 00:44:24.334551 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:24.339944 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Bootstrap starting.
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:24.342274 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:24.343298 2168 log.cc:824] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:24.345544 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: No bootstrap required, opened a new log
I20260927 00:44:24.346782 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.334198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.62:46821)
0927 00:44:24.334653 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:24.334675 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:24.334720 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:44:24.335363 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:24.335379 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:24.335707 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:24.336050 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:24.336826 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.337880 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.338747 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.339212 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.339384 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.341896 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:24.341901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:24.341915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:24.341943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:24.343817 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.344338 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.344349 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.344353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.344461 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.344735 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.344737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.344738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.345141 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:24.345280 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:24.345677 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 00:44:24.346041 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260927 00:44:24.349081 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260927 00:44:24.349303 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:24.349385 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36278c05cd6d4e159370c7bffa47069b, State: Initialized, Role: FOLLOWER
I20260927 00:44:24.349851 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260927 00:44:24.350006 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:24.350068 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:24.350162 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:24.351089 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260927 00:44:24.351429 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36278c05cd6d4e159370c7bffa47069b; no voters:
I20260927 00:44:24.351732 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:24.351857 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:24.352088 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [term 1 LEADER]: Becoming Leader. State: Replica: 36278c05cd6d4e159370c7bffa47069b, State: Running, Role: LEADER
I20260927 00:44:24.352434 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260927 00:44:24.352980 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:24.354130 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36278c05cd6d4e159370c7bffa47069b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } } }
I20260927 00:44:24.354265 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:24.354478 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36278c05cd6d4e159370c7bffa47069b. Latest consensus state: current_term: 1 leader_uuid: "36278c05cd6d4e159370c7bffa47069b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36278c05cd6d4e159370c7bffa47069b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } } }
I20260927 00:44:24.354619 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:24.354918 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:24.357524 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:24.362612 2182 catalog_manager.cc:1347] Generated new cluster ID: 4fbca10f66a746d19f6f0f51b80727ab
I20260927 00:44:24.362680 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:24.376813 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:24.377751 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:24.386852 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Generated new TSK 0
I20260927 00:44:24.387526 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:24.416613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:24.528023 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:24.528299 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:24.528355 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:24.532594 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:24.532658 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:24.532701 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:44:24.532714 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:24.532791 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:24.538335 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:24.539448 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:24.540505 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:24.550022 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.550235 2194 server_base.cc:1034] running on GCE node
W20260927 00:44:24.550012 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:24.550061 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.550786 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:24.551388 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:24.552539 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790469864552473 us; error 84 us; skew 500 ppm
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:24.555759 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:24.556641 2194 webserver.cc:460] Webserver started at http://127.0.98.1:36575/ using document root <none> and password file <none>
I20260927 00:44:24.557211 2194 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:24.557301 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:24.557534 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:24.559613 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "72447db69b78416681da6082ddfad3f8"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "36616e83cd448fd8e9570958c08b5122"
server_key_iv: "9e1b2537105c6772678a8b00d2abd475"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.560181 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72447db69b78416681da6082ddfad3f8"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "36616e83cd448fd8e9570958c08b5122"
server_key_iv: "9e1b2537105c6772678a8b00d2abd475"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.563931 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:24.566612 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:24.567986 2194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:24.568152 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "72447db69b78416681da6082ddfad3f8"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "36616e83cd448fd8e9570958c08b5122"
server_key_iv: "9e1b2537105c6772678a8b00d2abd475"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.568284 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:24.587119 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:24.590202 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:24.590427 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:24.591850 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:24.591934 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:24.592006 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:24.592048 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:24.602576 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38881
I20260927 00:44:24.602665 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38881 every 8 connection(s)
I20260927 00:44:24.603456 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:24.607221 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.592861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59353 (local address 127.0.98.62:46821)
0927 00:44:24.592997 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:24.593000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:24.594100 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 00:44:24.595246 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:24.595250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:24.595305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:24.595373 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:24.597157 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.598052 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.599863 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.600084 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.600169 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.603218 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:24.603221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:24.603224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:24.603252 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:24.604907 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.605538 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.605542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.605544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.605661 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.606105 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.606108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.606110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.606339 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:24.606469 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:24.606998 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 00:44:24.607105 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:24.608089 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.593358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.1:59353)
0927 00:44:24.593944 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:24.593987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:24.594993 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:24.595844 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:24.595853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:24.596295 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:24.597014 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.597036 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.598201 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.598207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.599729 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.599740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.600246 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.600251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.600479 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.601161 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:24.601188 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:24.603050 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:24.605026 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.605032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.605045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.605410 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.605786 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.605790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.605792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.605992 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.606464 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:24.606472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:24.606708 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 00:44:24.606936 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 00:44:24.607180 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":140,"threads_started":1}
I20260927 00:44:24.609309 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260927 00:44:24.609470 2325 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:24.609989 2325 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260927 00:44:24.611609 2121 ts_manager.cc:194] Registered new tserver with Master: 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881)
I20260927 00:44:24.612998 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59353
I20260927 00:44:24.613175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2194
I20260927 00:44:24.613284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:24.613577 392 external_mini_cluster.cc:1428] Setting key 1c4b44a9e76ea5f2c37d2372eaa17b08
I20260927 00:44:24.620210 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.611506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.1:59005)
0927 00:44:24.611808 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:24.611824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:24.611918 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:24.612140 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:24.612142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:24.612341 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:24.612454 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.612461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.613257 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.613263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.614110 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.614120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.614587 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.614591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.614681 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.616557 (+ 1876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:24.616569 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:24.616891 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:24.618736 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.618741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.618744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.618942 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.619224 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.619226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.619228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.619315 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.619667 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:24.619670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:24.619776 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 00:44:24.619889 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:44:24.620028 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
I20260927 00:44:24.620204 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.611618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59005 (local address 127.0.98.62:46821)
0927 00:44:24.611817 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:24.611821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:24.611894 (+ 73us) server_negotiation.cc:407] Connection header received
0927 00:44:24.612006 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:24.612009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:24.612045 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:24.612101 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:24.612562 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.613061 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.614209 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.614473 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.614587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.617024 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:24.617027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:24.617030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:24.617052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:24.618630 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.619038 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.619045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.619049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.619127 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.619403 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.619407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.619409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.619571 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:24.619677 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:24.619941 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 00:44:24.620047 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:24.639056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:24.748744 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:24.748983 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:24.749056 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:24.753152 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:24.753208 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:24.753275 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:44:24.753311 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:24.753374 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:24.758791 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:24.759824 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:24.760872 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:24.770144 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:24.770151 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:24.770153 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.770478 2334 server_base.cc:1034] running on GCE node
I20260927 00:44:24.770934 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:24.771762 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:24.772931 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790469864772903 us; error 48 us; skew 500 ppm
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:24.776851 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:24.778051 2334 webserver.cc:460] Webserver started at http://127.0.98.2:36257/ using document root <none> and password file <none>
I20260927 00:44:24.778689 2334 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:24.778753 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:24.778921 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:24.780864 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab7fc76f58cd46fb9325cafee0b16371"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "61c29ec1088bc98ae735a20fe51dad05"
server_key_iv: "9ed5ad979731ece8b68ad14414fbb7a7"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.781351 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab7fc76f58cd46fb9325cafee0b16371"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "61c29ec1088bc98ae735a20fe51dad05"
server_key_iv: "9ed5ad979731ece8b68ad14414fbb7a7"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.784930 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:24.787230 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:24.788414 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:24.788530 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "ab7fc76f58cd46fb9325cafee0b16371"
format_stamp: "Formatted at 2026-09-27 00:44:24 on dist-test-slave-l5sb"
server_key: "61c29ec1088bc98ae735a20fe51dad05"
server_key_iv: "9ed5ad979731ece8b68ad14414fbb7a7"
server_key_version: "encryptionkey@0"
I20260927 00:44:24.788627 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:24.811007 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:24.813891 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:24.814088 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:24.815586 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:24.815632 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:24.815697 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:24.815730 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:24 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:24.826658 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38963
I20260927 00:44:24.826822 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38963 every 8 connection(s)
I20260927 00:44:24.827566 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:24.829842 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.816476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56313 (local address 127.0.98.62:46821)
0927 00:44:24.816598 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:24.816601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:24.817673 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 00:44:24.818875 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:24.818882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:24.818954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:24.819057 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:24.820738 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.821646 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.822947 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.823348 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.823464 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.826127 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:24.826131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:24.826135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:24.826172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:24.827879 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.828469 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.828472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.828474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.828563 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.828883 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.828886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.828887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.829055 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:24.829128 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:24.829579 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 00:44:24.829675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:44:24.830606 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.817041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.2:56313)
0927 00:44:24.817531 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:24.817572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:24.818552 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:24.819332 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:24.819341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:24.819854 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:24.820574 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.820596 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.821814 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.821818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.822796 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.822805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.823555 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.823562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.823722 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.824323 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:24.824368 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:24.825959 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:24.827994 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.827999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.828013 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.828362 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.828659 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.828662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.828663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.828788 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.829138 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:24.829143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:24.829374 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 00:44:24.829519 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:44:24.829750 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":138,"threads_started":1}
I20260927 00:44:24.831647 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260927 00:44:24.831800 2465 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:24.832266 2465 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260927 00:44:24.833279 2122 ts_manager.cc:194] Registered new tserver with Master: ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963)
I20260927 00:44:24.834074 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56313
I20260927 00:44:24.835217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2334
I20260927 00:44:24.835350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:24.835624 392 external_mini_cluster.cc:1428] Setting key 4be8b4eb22a1e3a0cd1f8825cf37872f
I20260927 00:44:24.841455 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.833803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50291 (local address 127.0.98.62:46821)
0927 00:44:24.833903 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:24.833907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:24.834149 (+ 242us) server_negotiation.cc:407] Connection header received
0927 00:44:24.834319 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:24.834324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:24.834386 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:24.834468 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:24.834943 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.835647 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.836713 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.836986 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.837091 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.838215 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:24.838222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:24.838228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:24.838260 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:24.839669 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.840211 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.840214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.840216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.840302 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:24.840613 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:24.840615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:24.840616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:24.840820 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:24.840898 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:24.841227 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:44:24.841344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:44:24.841555 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:24.833679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.2:50291)
0927 00:44:24.834059 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:24.834073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:24.834174 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:24.834495 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:24.834502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:24.834704 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:24.834807 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.834813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.835797 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.835800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.836584 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:24.836592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:24.837097 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:24.837099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:24.837166 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:24.837684 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:24.837695 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:24.838087 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:24.839791 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.839797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.839802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.840089 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.840409 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:24.840415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:24.840420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:24.840523 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:24.840926 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:24.840931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:24.841043 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 00:44:24.841168 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 00:44:24.841330 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:24.865144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:24.975253 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:24.975517 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:24.975564 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:24.979488 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:24.979548 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:24.979589 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:44:24.979604 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:24.979633 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:24.985184 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:24.986693 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:24.988157 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:24.999202 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:24.999185 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.999231 2474 server_base.cc:1034] running on GCE node
W20260927 00:44:24.999190 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:24.999814 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:25.000473 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:25.001662 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790469865001640 us; error 32 us; skew 500 ppm
Sep 27 00:44:25 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:25.004562 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:25.005615 2474 webserver.cc:460] Webserver started at http://127.0.98.3:41649/ using document root <none> and password file <none>
I20260927 00:44:25.006210 2474 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:25.006274 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:25.006438 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:25.008618 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "a69bf07465e943feb0c496bd8f0d6184"
format_stamp: "Formatted at 2026-09-27 00:44:25 on dist-test-slave-l5sb"
server_key: "302d9fd49ff07fdfbe149fe7b2e1544f"
server_key_iv: "04b650c8698af11d0bf713fa53dfa275"
server_key_version: "encryptionkey@0"
I20260927 00:44:25.009227 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a69bf07465e943feb0c496bd8f0d6184"
format_stamp: "Formatted at 2026-09-27 00:44:25 on dist-test-slave-l5sb"
server_key: "302d9fd49ff07fdfbe149fe7b2e1544f"
server_key_iv: "04b650c8698af11d0bf713fa53dfa275"
server_key_version: "encryptionkey@0"
I20260927 00:44:25.012950 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:44:25.015301 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:25.016491 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:44:25.016610 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "a69bf07465e943feb0c496bd8f0d6184"
format_stamp: "Formatted at 2026-09-27 00:44:25 on dist-test-slave-l5sb"
server_key: "302d9fd49ff07fdfbe149fe7b2e1544f"
server_key_iv: "04b650c8698af11d0bf713fa53dfa275"
server_key_version: "encryptionkey@0"
I20260927 00:44:25.016705 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:25.034477 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:25.037330 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:25.037541 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:25.039055 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:25.039106 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:25.039170 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:25.039240 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:25 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:25.049587 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40721
I20260927 00:44:25.049981 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40721 every 8 connection(s)
I20260927 00:44:25.050547 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:25.051126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2474
I20260927 00:44:25.051230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:25.051556 392 external_mini_cluster.cc:1428] Setting key 1a07b5feb5da55f5943eb5cd98cb7e65
I20260927 00:44:25.053422 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.039883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46025 (local address 127.0.98.62:46821)
0927 00:44:25.040038 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.040045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.040820 (+ 775us) server_negotiation.cc:407] Connection header received
0927 00:44:25.041832 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.041836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.041891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.041980 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:25.043503 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.044405 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.046154 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.046405 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.046523 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.049347 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:25.049351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:25.049354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:25.049380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:25.051225 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.051815 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.051819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.051828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.051942 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.052313 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.052318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.052320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.052536 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:25.052642 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.053109 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.053226 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:25.054272 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.040243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.3:46025)
0927 00:44:25.040679 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.040727 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.041601 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.042152 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.042162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.042685 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:25.043322 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.043343 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.044552 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.044555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.046044 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.046055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.046568 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.046572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.046756 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.047345 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:25.047362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:25.049140 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:25.051359 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.051364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.051380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.051706 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.052067 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.052070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.052072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.052213 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.052640 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:25.052647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:25.052883 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.053068 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.053398 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":138,"threads_started":1}
I20260927 00:44:25.055377 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260927 00:44:25.055516 2605 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:25.056023 2605 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260927 00:44:25.057013 2122 ts_manager.cc:194] Registered new tserver with Master: a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:25.057750 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46025
I20260927 00:44:25.064978 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.057492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48471 (local address 127.0.98.62:46821)
0927 00:44:25.057629 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.057633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.057827 (+ 194us) server_negotiation.cc:407] Connection header received
0927 00:44:25.057989 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.057991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.058056 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.058136 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:25.058675 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.059413 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.060357 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.060566 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.060648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.061739 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:25.061742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:25.061744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:25.061765 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:25.063259 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.063646 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.063648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.063650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.063774 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.064074 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.064077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.064079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.064293 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:25.064421 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.064714 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.064847 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:44:25.064990 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.057425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.3:48471)
0927 00:44:25.057728 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.057744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.057899 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.058168 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.058171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.058435 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:25.058556 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.058562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.059570 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.059572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.060228 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.060237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.060691 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.060694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.060785 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.061260 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:25.061272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:25.061611 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:25.063365 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.063367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.063369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.063561 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.063861 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.063863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.063864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.063966 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.064421 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:25.064424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:25.064531 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.064632 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.064747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260927 00:44:25.065557 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:25 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:44:25 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:25.091013 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.083364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.98.62:46821)
0927 00:44:25.083492 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.083496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.083509 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:25.083552 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.083554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.083595 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.083669 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:25.084319 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.084826 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.085609 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.085864 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.085943 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.088141 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:25.088144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:25.088146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:25.088169 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:25.089564 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.090019 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.090022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.090024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.090101 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.090344 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.090345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.090346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.090533 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:25.090631 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.090794 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.090889 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:44:25.093892 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:25.096740 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:25.111244 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.105906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.98.1:38881)
0927 00:44:25.106568 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.106589 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.106607 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:44:25.106688 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.106695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.106889 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.107063 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.107258 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108143 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110004 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110600 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.110877 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.111010 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.111102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":90,"threads_started":1}
I20260927 00:44:25.111275 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.104993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38963 (local address 127.0.0.1:57872)
0927 00:44:25.105341 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.105400 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.105521 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.106733 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.106739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.106762 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.106896 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.106904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108388 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.108391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.109728 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.109738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110793 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.110883 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.110906 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.110963 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.111100 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260927 00:44:25.111258 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.105719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.0.1:32850)
0927 00:44:25.106041 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.106054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.106353 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.107034 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.107036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.107052 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.107146 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.107151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108252 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.108258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.109886 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.109893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110721 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.110834 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.110870 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.110982 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.111104 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":313,"thread_start_us":127,"threads_started":1}
I20260927 00:44:25.111577 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.105812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.0.98.3:40721)
0927 00:44:25.106249 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.106262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.106285 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:25.106362 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.106368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.106571 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.106733 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.107151 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108277 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.109723 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110648 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110772 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.110925 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.111326 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.111442 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":110,"threads_started":1}
I20260927 00:44:25.111858 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.105283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.0.1:51796)
0927 00:44:25.105556 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.105570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.106153 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.106890 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.106893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.106907 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.107019 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.107025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108437 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.108440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.109595 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.109604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110983 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.110986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.111071 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.111090 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.111157 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.111214 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 00:44:25.112026 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.105121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.0.98.2:38963)
0927 00:44:25.106249 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.106262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.106285 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:25.106363 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.106368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.106571 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.106732 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.107024 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.108277 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.109877 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.110655 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.111505 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.111662 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.111718 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.111871 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":98,"threads_started":1}
I20260927 00:44:25.114491 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 6b13b8cc902845f7b61409a932aee897 (DEFAULT_TABLE table=test-table [id=503e3170672a47338fa014ea57375f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.114495 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 6b13b8cc902845f7b61409a932aee897 (DEFAULT_TABLE table=test-table [id=503e3170672a47338fa014ea57375f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.114672 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 6b13b8cc902845f7b61409a932aee897 (DEFAULT_TABLE table=test-table [id=503e3170672a47338fa014ea57375f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.115514 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b13b8cc902845f7b61409a932aee897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.115540 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b13b8cc902845f7b61409a932aee897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.115547 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b13b8cc902845f7b61409a932aee897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.122179 2628 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Bootstrap starting.
I20260927 00:44:25.123656 2629 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Bootstrap starting.
I20260927 00:44:25.124766 2628 tablet_bootstrap.cc:654] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.125577 2629 tablet_bootstrap.cc:654] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.125597 2628 log.cc:824] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:25.126098 2630 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Bootstrap starting.
I20260927 00:44:25.126784 2629 log.cc:824] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:25.127648 2628 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: No bootstrap required, opened a new log
I20260927 00:44:25.127835 2628 ts_tablet_manager.cc:1397] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:25.128048 2630 tablet_bootstrap.cc:654] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.128590 2629 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: No bootstrap required, opened a new log
I20260927 00:44:25.128690 2630 log.cc:824] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:25.128767 2629 ts_tablet_manager.cc:1397] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:25.130340 2630 tablet_bootstrap.cc:492] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: No bootstrap required, opened a new log
I20260927 00:44:25.130573 2630 ts_tablet_manager.cc:1397] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:25.131662 2628 raft_consensus.cc:348] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.131935 2628 raft_consensus.cc:374] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.132018 2628 raft_consensus.cc:729] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.132479 2628 consensus_queue.cc:260] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.132577 2629 raft_consensus.cc:348] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.132886 2629 raft_consensus.cc:374] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.132974 2629 raft_consensus.cc:729] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.133158 2465 heartbeater.cc:503] Master 127.0.98.62:46821 was elected leader, sending a full tablet report...
I20260927 00:44:25.133409 2629 consensus_queue.cc:260] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.133528 2628 ts_tablet_manager.cc:1428] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:44:25.134040 2325 heartbeater.cc:503] Master 127.0.98.62:46821 was elected leader, sending a full tablet report...
I20260927 00:44:25.134371 2629 ts_tablet_manager.cc:1428] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:44:25.134999 2630 raft_consensus.cc:348] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.135237 2630 raft_consensus.cc:374] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.135304 2630 raft_consensus.cc:729] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.135691 2630 consensus_queue.cc:260] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.136364 2605 heartbeater.cc:503] Master 127.0.98.62:46821 was elected leader, sending a full tablet report...
I20260927 00:44:25.136641 2630 ts_tablet_manager.cc:1428] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260927 00:44:25.302078 2606 tablet.cc:2369] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:25.329092 2466 tablet.cc:2369] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:25.355235 2326 tablet.cc:2369] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:25.359207 2634 raft_consensus.cc:482] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:25.359438 2634 raft_consensus.cc:504] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.360594 2634 leader_election.cc:288] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:25.364428 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.360971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40645 (local address 127.0.98.1:38881)
0927 00:44:25.361098 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.361103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.361271 (+ 168us) server_negotiation.cc:407] Connection header received
0927 00:44:25.361436 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.361439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.361490 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.361575 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.361897 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.362440 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.363481 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.363878 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.363995 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.364057 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.364242 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.364318 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:25.364440 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.360869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.98.2:40645)
0927 00:44:25.361171 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.361183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.361318 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.361587 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.361590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.361607 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.361738 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.361743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.362558 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.362563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.363354 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.363362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.364011 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.364014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.364110 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.364132 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.364227 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.364295 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260927 00:44:25.364538 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.361002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41957 (local address 127.0.98.3:40721)
0927 00:44:25.361130 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.361136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.361490 (+ 354us) server_negotiation.cc:407] Connection header received
0927 00:44:25.361641 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.361644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.361693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.361771 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.362107 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.362620 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.363599 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.363997 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.364111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.364167 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.364346 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.364411 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:44:25.364603 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.360869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.98.2:41957)
0927 00:44:25.361387 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.361402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.361529 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.361864 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.361868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.361888 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.361993 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.361998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.362744 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.362749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.363474 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.363482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.364115 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.364118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.364214 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.364234 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.364332 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.364398 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":287,"thread_start_us":93,"threads_started":1}
I20260927 00:44:25.365198 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b13b8cc902845f7b61409a932aee897" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184" is_pre_election: true
I20260927 00:44:25.365193 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b13b8cc902845f7b61409a932aee897" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8" is_pre_election: true
I20260927 00:44:25.365475 2279 raft_consensus.cc:2454] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 0.
I20260927 00:44:25.365482 2559 raft_consensus.cc:2454] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 0.
I20260927 00:44:25.365983 2353 leader_election.cc:302] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69bf07465e943feb0c496bd8f0d6184, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:25.366259 2634 raft_consensus.cc:2781] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:25.366343 2634 raft_consensus.cc:482] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:25.366385 2634 raft_consensus.cc:3037] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.367408 2634 raft_consensus.cc:504] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.367717 2634 leader_election.cc:288] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 election: Requested vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:25.368024 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b13b8cc902845f7b61409a932aee897" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8"
I20260927 00:44:25.368052 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b13b8cc902845f7b61409a932aee897" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184"
I20260927 00:44:25.368165 2279 raft_consensus.cc:3037] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.368180 2559 raft_consensus.cc:3037] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.369210 2279 raft_consensus.cc:2454] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 1.
I20260927 00:44:25.369505 2351 leader_election.cc:302] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:25.369554 2559 raft_consensus.cc:2454] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 1.
I20260927 00:44:25.369679 2634 raft_consensus.cc:2781] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:25.369890 2634 raft_consensus.cc:686] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 LEADER]: Becoming Leader. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Running, Role: LEADER
I20260927 00:44:25.370217 2634 consensus_queue.cc:237] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.373324 2121 catalog_manager.cc:5495] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ab7fc76f58cd46fb9325cafee0b16371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:44:25 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:25.432022 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.423554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.0.98.62:46821)
0927 00:44:25.423682 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.423687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.423873 (+ 186us) server_negotiation.cc:407] Connection header received
0927 00:44:25.424022 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.424026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.424070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.424155 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:25.424829 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.425342 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.426197 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.426410 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.426512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.428871 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:25.428875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:25.428877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:25.428896 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:25.430298 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.430759 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.430765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.430766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.430844 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:25.431105 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:25.431110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:25.431112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:25.431277 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:25.431380 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.431662 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.431855 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":44}
I20260927 00:44:25.432022 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.423469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.0.1:53848)
0927 00:44:25.423767 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.423779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.423918 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.424174 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.424177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.424615 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:25.424702 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.424708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.425447 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.425453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.426082 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.426092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.426525 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.426528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.426623 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.427228 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:25.427241 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:25.428722 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:25.430418 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.430426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.430433 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.430654 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.430931 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:25.430935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:25.430938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:25.431017 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:25.431381 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:25.431389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:25.431500 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.431617 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.431819 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260927 00:44:25.435952 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:44:25.436692 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:25.441331 2399 tablet_service.cc:1467] Processing CreateTablet for tablet e85abc86331a4375b333226d2d95004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40f71eb658ed4273962f3867243e9cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:25.441362 2539 tablet_service.cc:1467] Processing CreateTablet for tablet e85abc86331a4375b333226d2d95004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40f71eb658ed4273962f3867243e9cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:25.441270 2259 tablet_service.cc:1467] Processing CreateTablet for tablet e85abc86331a4375b333226d2d95004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40f71eb658ed4273962f3867243e9cf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:25.442001 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85abc86331a4375b333226d2d95004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.442001 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85abc86331a4375b333226d2d95004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.442009 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85abc86331a4375b333226d2d95004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.444337 2628 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: Bootstrap starting.
I20260927 00:44:25.444756 2630 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: Bootstrap starting.
I20260927 00:44:25.445729 2628 tablet_bootstrap.cc:654] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.445729 2630 tablet_bootstrap.cc:654] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.446417 2629 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: Bootstrap starting.
I20260927 00:44:25.447073 2630 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: No bootstrap required, opened a new log
I20260927 00:44:25.447157 2628 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: No bootstrap required, opened a new log
I20260927 00:44:25.447154 2630 ts_tablet_manager.cc:1397] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:25.447223 2628 ts_tablet_manager.cc:1397] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:44:25.447341 2629 tablet_bootstrap.cc:654] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.447616 2628 raft_consensus.cc:348] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.447616 2630 raft_consensus.cc:348] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.447722 2628 raft_consensus.cc:374] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.447742 2630 raft_consensus.cc:374] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.447759 2628 raft_consensus.cc:729] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.447779 2630 raft_consensus.cc:729] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.447902 2630 consensus_queue.cc:260] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.447901 2628 consensus_queue.cc:260] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.448166 2628 ts_tablet_manager.cc:1428] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:25.448127 2634 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.448237 2634 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.448575 2630 ts_tablet_manager.cc:1428] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:25.448586 2636 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.448789 2636 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.449479 2629 tablet_bootstrap.cc:492] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: No bootstrap required, opened a new log
I20260927 00:44:25.449584 2629 ts_tablet_manager.cc:1397] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:44:25.450165 2629 raft_consensus.cc:348] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.450281 2629 raft_consensus.cc:374] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.450306 2629 raft_consensus.cc:729] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.450434 2629 consensus_queue.cc:260] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.450696 2629 ts_tablet_manager.cc:1428] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:25.450805 2635 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.450973 2635 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.520390 2636 raft_consensus.cc:482] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:25.520558 2636 raft_consensus.cc:504] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.521644 2636 leader_election.cc:288] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963)
I20260927 00:44:25.525673 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.522046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39647 (local address 127.0.98.1:38881)
0927 00:44:25.522194 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.522199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.522215 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:44:25.522362 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.522366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.522417 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.522493 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.522871 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.523746 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.524746 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.525139 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.525256 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.525315 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.525488 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.525559 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:44:25.525676 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.521943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.98.3:39647)
0927 00:44:25.522089 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.522107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.522245 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.522535 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.522537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.522557 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.522741 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.522748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.523853 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.523856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.524626 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.524633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.525256 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.525260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.525371 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.525388 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.525474 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.525543 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3,"spinlock_wait_cycles":12928}
I20260927 00:44:25.525918 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.522212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38963 (local address 127.0.98.3:52299)
0927 00:44:25.522515 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.522529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.522645 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.522912 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.522914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.522929 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:25.523047 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.523053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.524013 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.524019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.524824 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.524832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.525484 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.525487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.525578 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.525598 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.525693 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.525756 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"spinlock_wait_cycles":768,"thread_start_us":74,"threads_started":1}
I20260927 00:44:25.525914 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.522335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52299 (local address 127.0.98.2:38963)
0927 00:44:25.522496 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.522502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.522604 (+ 102us) server_negotiation.cc:407] Connection header received
0927 00:44:25.522759 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.522762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.522825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.522906 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:25.523179 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.523898 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.524952 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.525350 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.525469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.525530 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.525709 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.525802 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:44:25.526347 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85abc86331a4375b333226d2d95004b" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8" is_pre_election: true
I20260927 00:44:25.526499 2279 raft_consensus.cc:2454] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 0.
I20260927 00:44:25.526500 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85abc86331a4375b333226d2d95004b" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371" is_pre_election: true
I20260927 00:44:25.526692 2419 raft_consensus.cc:2454] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 0.
I20260927 00:44:25.526841 2491 leader_election.cc:302] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, a69bf07465e943feb0c496bd8f0d6184; no voters:
I20260927 00:44:25.527153 2636 raft_consensus.cc:2781] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:25.527226 2636 raft_consensus.cc:482] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:25.527279 2636 raft_consensus.cc:3037] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.528218 2636 raft_consensus.cc:504] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.528553 2636 leader_election.cc:288] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 election: Requested vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963)
I20260927 00:44:25.528878 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85abc86331a4375b333226d2d95004b" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8"
I20260927 00:44:25.528991 2279 raft_consensus.cc:3037] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.528941 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85abc86331a4375b333226d2d95004b" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371"
I20260927 00:44:25.529091 2419 raft_consensus.cc:3037] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.530030 2279 raft_consensus.cc:2454] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 1.
I20260927 00:44:25.530236 2419 raft_consensus.cc:2454] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 1.
I20260927 00:44:25.530290 2491 leader_election.cc:302] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, a69bf07465e943feb0c496bd8f0d6184; no voters:
I20260927 00:44:25.530467 2636 raft_consensus.cc:2781] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:25.530674 2636 raft_consensus.cc:686] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [term 1 LEADER]: Becoming Leader. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Running, Role: LEADER
I20260927 00:44:25.531059 2636 consensus_queue.cc:237] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.533226 2651 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: TxnStatusTablet state changed. Reason: New leader a69bf07465e943feb0c496bd8f0d6184. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.533347 2651 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:25.533615 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:44:25.533854 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:44:25.534466 2121 catalog_manager.cc:5495] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } health_report { overall_health: HEALTHY } } }
I20260927 00:44:25.552093 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.548292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.0.1:46144)
0927 00:44:25.548440 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.548464 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.548547 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.548851 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.548853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.548858 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:25.548941 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.548945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.549938 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.549941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.550537 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.550545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.551086 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.551088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.551147 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.551187 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.551869 (+ 682us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.551878 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.551925 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.551987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:44:25.552117 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.548378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.98.3:40721)
0927 00:44:25.548499 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.548504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.548518 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:25.548671 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.548674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.548733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.548834 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:25.549035 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.549837 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.550647 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.550985 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.551083 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.551312 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.551782 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.551889 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.551943 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.551993 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:44:25.557185 2279 raft_consensus.cc:1261] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Refusing update from remote peer a69bf07465e943feb0c496bd8f0d6184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.557184 2419 raft_consensus.cc:1261] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Refusing update from remote peer a69bf07465e943feb0c496bd8f0d6184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.557855 2636 consensus_queue.cc:1035] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:25.558094 2651 consensus_queue.cc:1035] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:25.560925 2634 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: TxnStatusTablet state changed. Reason: New leader a69bf07465e943feb0c496bd8f0d6184. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.561034 2634 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.561553 2635 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: TxnStatusTablet state changed. Reason: New leader a69bf07465e943feb0c496bd8f0d6184. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.561654 2635 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.565220 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333764569315971072 to 7333764569217986560. Current Snapshot: MvccSnapshot[applied={T|T < 7333764569315971072}]
I20260927 00:44:25.565376 2636 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.565467 2636 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:25.565689 2651 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.565858 2651 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P a69bf07465e943feb0c496bd8f0d6184: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:25.566869 2635 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.566972 2635 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P 72447db69b78416681da6082ddfad3f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.566957 2634 tablet_replica.cc:439] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } } }
I20260927 00:44:25.567054 2634 tablet_replica.cc:442] T e85abc86331a4375b333226d2d95004b P ab7fc76f58cd46fb9325cafee0b16371: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:25.567553 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333764569315971072 to 7333764569217986560. Current Snapshot: MvccSnapshot[applied={T|T < 7333764569315971072}]
I20260927 00:44:25.577503 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.573630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.0.98.2:38963)
0927 00:44:25.573746 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.573750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.573824 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:44:25.573965 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.573967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.574012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.574092 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:25.574342 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.574846 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.576126 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.576343 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.576439 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.576741 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.577116 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.577206 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.577294 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.577342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:44:25.588007 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.584647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59617 (local address 127.0.98.3:40721)
0927 00:44:25.584778 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.584783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.585023 (+ 240us) server_negotiation.cc:407] Connection header received
0927 00:44:25.585167 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.585170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.585216 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.585303 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:25.585551 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.586119 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.586981 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.587200 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.587302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.587595 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.587665 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.587741 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.587832 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.587875 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:25.588045 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.584573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.98.2:59617)
0927 00:44:25.584942 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.584954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.585064 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.585318 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.585321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.585328 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:25.585449 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.585454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.586234 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.586237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.586857 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.586865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.587304 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.587314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.587410 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.587446 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.587756 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.587762 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.587822 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.587894 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":113,"threads_started":1}
I20260927 00:44:25.594190 2279 raft_consensus.cc:1261] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Refusing update from remote peer ab7fc76f58cd46fb9325cafee0b16371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.594256 2559 raft_consensus.cc:1261] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Refusing update from remote peer ab7fc76f58cd46fb9325cafee0b16371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.594609 2639 consensus_queue.cc:1035] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:25.594760 2634 consensus_queue.cc:1035] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:25.603564 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333764569469091840 to 7333764568558759936. Current Snapshot: MvccSnapshot[applied={T|T < 7333764569469091840}]
I20260927 00:44:25.621934 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.615086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33485 (local address 127.0.98.2:38963)
0927 00:44:25.615193 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:25.615197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:25.618089 (+ 2892us) server_negotiation.cc:407] Connection header received
0927 00:44:25.618189 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:25.618192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:25.618241 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:25.618322 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:25.618634 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.619431 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.620696 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.621025 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.621093 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.621471 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.621549 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.621590 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:25.621734 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:44:25.621802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:44:25.622488 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:25.614900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38963 (local address 127.0.98.3:33485)
0927 00:44:25.617988 (+ 3088us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:25.618003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:25.618106 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:25.618339 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:25.618349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:25.618368 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:25.618492 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.618498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.619558 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.619561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.620564 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:25.620574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:25.621196 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:25.621199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:25.621289 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:25.621343 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:25.621646 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:25.621651 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:44:25.622242 (+ 591us) client_negotiation.cc:240] Negotiation successful
0927 00:44:25.622331 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2974,"thread_start_us":110,"threads_started":1}
I20260927 00:44:25.648025 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:25.648211 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:25.650501 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Sending DeleteTablet for 3 replicas of tablet 6b13b8cc902845f7b61409a932aee897
I20260927 00:44:25.651386 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b13b8cc902845f7b61409a932aee897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32850
I20260927 00:44:25.651526 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b13b8cc902845f7b61409a932aee897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57872
I20260927 00:44:25.651902 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b13b8cc902845f7b61409a932aee897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260927 00:44:25.652056 2679 tablet_replica.cc:330] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: stopping tablet replica
I20260927 00:44:25.652374 2679 raft_consensus.cc:2229] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:25.652834 2679 raft_consensus.cc:2258] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:25.653041 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:25.653486 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:25.654656 2678 tablet_replica.cc:330] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: stopping tablet replica
I20260927 00:44:25.655030 2678 raft_consensus.cc:2229] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:25.655232 2679 ts_tablet_manager.cc:1905] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:25.655485 2678 raft_consensus.cc:2258] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:25.658269 2259 tablet_service.cc:1467] Processing CreateTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (DEFAULT_TABLE table=test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.658547 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9132866d00a44a8a3ae28c6bc3dd3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.659147 2678 ts_tablet_manager.cc:1905] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:25.660928 2629 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Bootstrap starting.
I20260927 00:44:25.661986 2679 ts_tablet_manager.cc:1918] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:25.662724 2539 tablet_service.cc:1467] Processing CreateTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (DEFAULT_TABLE table=test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.662835 2629 tablet_bootstrap.cc:654] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.663003 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9132866d00a44a8a3ae28c6bc3dd3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.664284 2678 ts_tablet_manager.cc:1918] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:25.662417 2399 tablet_service.cc:1467] Processing CreateTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (DEFAULT_TABLE table=test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:25.664575 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9132866d00a44a8a3ae28c6bc3dd3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:25.666507 2630 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Bootstrap starting.
I20260927 00:44:25.666863 2628 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Bootstrap starting.
I20260927 00:44:25.667476 2630 tablet_bootstrap.cc:654] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.667781 2628 tablet_bootstrap.cc:654] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:25.668730 2679 log.cc:1196] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/6b13b8cc902845f7b61409a932aee897
I20260927 00:44:25.669143 2679 ts_tablet_manager.cc:1939] T 6b13b8cc902845f7b61409a932aee897 P ab7fc76f58cd46fb9325cafee0b16371: Deleting consensus metadata
I20260927 00:44:25.669795 2630 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: No bootstrap required, opened a new log
I20260927 00:44:25.669884 2630 ts_tablet_manager.cc:1397] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:25.669994 2678 log.cc:1196] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/6b13b8cc902845f7b61409a932aee897
I20260927 00:44:25.670361 2678 ts_tablet_manager.cc:1939] T 6b13b8cc902845f7b61409a932aee897 P 72447db69b78416681da6082ddfad3f8: Deleting consensus metadata
I20260927 00:44:25.670424 2630 raft_consensus.cc:348] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.670539 2630 raft_consensus.cc:374] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.670578 2630 raft_consensus.cc:729] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.670717 2630 consensus_queue.cc:260] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.671018 2630 ts_tablet_manager.cc:1428] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:25.672055 2107 catalog_manager.cc:4841] TS 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881): tablet 6b13b8cc902845f7b61409a932aee897 (table test-table [id=503e3170672a47338fa014ea57375f80]) successfully deleted
I20260927 00:44:25.672565 2107 catalog_manager.cc:4841] TS ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963): tablet 6b13b8cc902845f7b61409a932aee897 (table test-table [id=503e3170672a47338fa014ea57375f80]) successfully deleted
I20260927 00:44:25.673549 2628 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: No bootstrap required, opened a new log
I20260927 00:44:25.673632 2628 ts_tablet_manager.cc:1397] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 00:44:25.673686 2680 tablet_replica.cc:330] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: stopping tablet replica
I20260927 00:44:25.673985 2680 raft_consensus.cc:2229] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:25.674125 2680 raft_consensus.cc:2258] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:25.674194 2628 raft_consensus.cc:348] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.674295 2628 raft_consensus.cc:374] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.674325 2628 raft_consensus.cc:729] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.674474 2628 consensus_queue.cc:260] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.674734 2628 ts_tablet_manager.cc:1428] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:25.676177 2680 ts_tablet_manager.cc:1905] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:25.677474 2629 tablet_bootstrap.cc:492] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: No bootstrap required, opened a new log
I20260927 00:44:25.677567 2629 ts_tablet_manager.cc:1397] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260927 00:44:25.678095 2629 raft_consensus.cc:348] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.678215 2629 raft_consensus.cc:374] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:25.678259 2629 raft_consensus.cc:729] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:25.678380 2629 consensus_queue.cc:260] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.678661 2629 ts_tablet_manager.cc:1428] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:25.681406 2680 ts_tablet_manager.cc:1918] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:25.681550 2680 log.cc:1196] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/6b13b8cc902845f7b61409a932aee897
I20260927 00:44:25.681936 2680 ts_tablet_manager.cc:1939] T 6b13b8cc902845f7b61409a932aee897 P a69bf07465e943feb0c496bd8f0d6184: Deleting consensus metadata
I20260927 00:44:25.682963 2109 catalog_manager.cc:4841] TS a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721): tablet 6b13b8cc902845f7b61409a932aee897 (table test-table [id=503e3170672a47338fa014ea57375f80]) successfully deleted
I20260927 00:44:25.770174 2634 raft_consensus.cc:482] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:25.770334 2634 raft_consensus.cc:504] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.770788 2634 leader_election.cc:288] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:25.771230 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9132866d00a44a8a3ae28c6bc3dd3f5" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8" is_pre_election: true
I20260927 00:44:25.771414 2279 raft_consensus.cc:2454] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 0.
I20260927 00:44:25.771416 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9132866d00a44a8a3ae28c6bc3dd3f5" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184" is_pre_election: true
I20260927 00:44:25.771561 2559 raft_consensus.cc:2454] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 0.
I20260927 00:44:25.771781 2351 leader_election.cc:302] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:25.771971 2634 raft_consensus.cc:2781] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:25.772035 2634 raft_consensus.cc:482] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:25.772086 2634 raft_consensus.cc:3037] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.773240 2634 raft_consensus.cc:504] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.773617 2634 leader_election.cc:288] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 election: Requested vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:25.773928 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9132866d00a44a8a3ae28c6bc3dd3f5" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8"
I20260927 00:44:25.773977 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9132866d00a44a8a3ae28c6bc3dd3f5" candidate_uuid: "ab7fc76f58cd46fb9325cafee0b16371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184"
I20260927 00:44:25.774066 2279 raft_consensus.cc:3037] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.774130 2559 raft_consensus.cc:3037] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:25.775005 2559 raft_consensus.cc:2454] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 1.
I20260927 00:44:25.775223 2279 raft_consensus.cc:2454] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7fc76f58cd46fb9325cafee0b16371 in term 1.
I20260927 00:44:25.775314 2353 leader_election.cc:302] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69bf07465e943feb0c496bd8f0d6184, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:25.775483 2634 raft_consensus.cc:2781] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:25.775590 2634 raft_consensus.cc:686] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 LEADER]: Becoming Leader. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Running, Role: LEADER
I20260927 00:44:25.775732 2634 consensus_queue.cc:237] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:25.777537 2121 catalog_manager.cc:5495] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ab7fc76f58cd46fb9325cafee0b16371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:25.805348 2559 raft_consensus.cc:1261] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Refusing update from remote peer ab7fc76f58cd46fb9325cafee0b16371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.805827 2634 consensus_queue.cc:1035] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:25.806133 2279 raft_consensus.cc:1261] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Refusing update from remote peer ab7fc76f58cd46fb9325cafee0b16371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:25.806528 2634 consensus_queue.cc:1035] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:44:25.821650 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:44:27.372417 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:27.372555 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:27.374828 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Sending DeleteTablet for 3 replicas of tablet d9132866d00a44a8a3ae28c6bc3dd3f5
I20260927 00:44:27.375375 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32850
I20260927 00:44:27.375397 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260927 00:44:27.375468 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet d9132866d00a44a8a3ae28c6bc3dd3f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57872
I20260927 00:44:27.375936 2710 tablet_replica.cc:330] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: stopping tablet replica
I20260927 00:44:27.376088 2710 raft_consensus.cc:2229] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:27.376216 2710 raft_consensus.cc:2258] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.376758 2710 ts_tablet_manager.cc:1905] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.375703 2711 tablet_replica.cc:330] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: stopping tablet replica
I20260927 00:44:27.377629 2711 raft_consensus.cc:2229] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:27.377847 2711 raft_consensus.cc:2258] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.378381 2712 tablet_replica.cc:330] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: stopping tablet replica
I20260927 00:44:27.378504 2712 raft_consensus.cc:2229] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:27.378561 2711 ts_tablet_manager.cc:1905] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.378708 2712 raft_consensus.cc:2258] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.379318 2712 ts_tablet_manager.cc:1905] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.379537 2710 ts_tablet_manager.cc:1918] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:44:27.379624 2710 log.cc:1196] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/d9132866d00a44a8a3ae28c6bc3dd3f5
I20260927 00:44:27.379912 2710 ts_tablet_manager.cc:1939] T d9132866d00a44a8a3ae28c6bc3dd3f5 P 72447db69b78416681da6082ddfad3f8: Deleting consensus metadata
I20260927 00:44:27.380527 2107 catalog_manager.cc:4841] TS 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881): tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (table test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]) successfully deleted
I20260927 00:44:27.382217 2712 ts_tablet_manager.cc:1918] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:44:27.382296 2712 log.cc:1196] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/d9132866d00a44a8a3ae28c6bc3dd3f5
I20260927 00:44:27.382632 2712 ts_tablet_manager.cc:1939] T d9132866d00a44a8a3ae28c6bc3dd3f5 P ab7fc76f58cd46fb9325cafee0b16371: Deleting consensus metadata
I20260927 00:44:27.383306 2107 catalog_manager.cc:4841] TS ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963): tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (table test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]) successfully deleted
I20260927 00:44:27.384080 2711 ts_tablet_manager.cc:1918] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:44:27.384168 2711 log.cc:1196] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/d9132866d00a44a8a3ae28c6bc3dd3f5
I20260927 00:44:27.384454 2711 ts_tablet_manager.cc:1939] T d9132866d00a44a8a3ae28c6bc3dd3f5 P a69bf07465e943feb0c496bd8f0d6184: Deleting consensus metadata
I20260927 00:44:27.385035 2109 catalog_manager.cc:4841] TS a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721): tablet d9132866d00a44a8a3ae28c6bc3dd3f5 (table test-table [id=899b7bdc3a8b4441af32fac4fc835a2b]) successfully deleted
Sep 27 00:44:27 dist-test-slave-l5sb krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:44:27 dist-test-slave-l5sb krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:27.404116 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.395150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.0.98.62:46821)
0927 00:44:27.395434 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.395444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.395544 (+ 100us) server_negotiation.cc:407] Connection header received
0927 00:44:27.395724 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.395732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.395778 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.395878 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:27.396570 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.397152 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.398065 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.398302 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.398412 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.400954 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:27.400957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:27.400959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:27.400983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:27.402465 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:27.402886 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:27.402891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:27.402893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:27.402977 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:27.403240 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:27.403245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:27.403246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:27.403426 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:27.403565 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.403809 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.403940 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260927 00:44:27.406792 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:27.407182 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:27.412082 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 (DEFAULT_TABLE table=test-table [id=c03b18a7f8d44e098d93668f5306ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.412174 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 (DEFAULT_TABLE table=test-table [id=c03b18a7f8d44e098d93668f5306ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.412169 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 (DEFAULT_TABLE table=test-table [id=c03b18a7f8d44e098d93668f5306ea33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.412416 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723cd42e43ba4675bcfc0eccced43fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.412416 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723cd42e43ba4675bcfc0eccced43fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.412504 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723cd42e43ba4675bcfc0eccced43fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.414947 2721 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Bootstrap starting.
I20260927 00:44:27.415428 2723 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Bootstrap starting.
I20260927 00:44:27.415910 2721 tablet_bootstrap.cc:654] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.416299 2723 tablet_bootstrap.cc:654] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.416898 2722 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Bootstrap starting.
I20260927 00:44:27.417188 2721 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: No bootstrap required, opened a new log
I20260927 00:44:27.417282 2721 ts_tablet_manager.cc:1397] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:27.417819 2722 tablet_bootstrap.cc:654] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.417754 2721 raft_consensus.cc:348] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.417959 2721 raft_consensus.cc:374] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.418010 2721 raft_consensus.cc:729] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.418172 2721 consensus_queue.cc:260] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.418386 2723 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: No bootstrap required, opened a new log
I20260927 00:44:27.418507 2721 ts_tablet_manager.cc:1428] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:27.418519 2723 ts_tablet_manager.cc:1397] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:27.419003 2723 raft_consensus.cc:348] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.419111 2722 tablet_bootstrap.cc:492] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: No bootstrap required, opened a new log
I20260927 00:44:27.419138 2723 raft_consensus.cc:374] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.419173 2723 raft_consensus.cc:729] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.419207 2722 ts_tablet_manager.cc:1397] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:27.419358 2723 consensus_queue.cc:260] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.419659 2722 raft_consensus.cc:348] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.419802 2723 ts_tablet_manager.cc:1428] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:27.419824 2722 raft_consensus.cc:374] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.419868 2722 raft_consensus.cc:729] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.420004 2722 consensus_queue.cc:260] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.421603 2722 ts_tablet_manager.cc:1428] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:27.511947 2727 raft_consensus.cc:482] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:27.512127 2727 raft_consensus.cc:504] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.513442 2727 leader_election.cc:288] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:27.518596 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.513646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38963 (local address 127.0.98.1:55293)
0927 00:44:27.514070 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:27.514085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:27.514195 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:27.514558 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:27.514561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:27.514599 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:27.514828 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.514837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.515850 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.515853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.517114 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.517124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.518121 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.518124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.518190 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.518240 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:44:27.518315 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:44:27.518393 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":59,"spinlock_wait_cycles":7424,"thread_start_us":179,"threads_started":1}
I20260927 00:44:27.518639 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.513724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55293 (local address 127.0.98.2:38963)
0927 00:44:27.514109 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.514118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.514141 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:27.514362 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.514366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.514424 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.514535 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:27.514956 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.515735 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.517254 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.517972 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.518085 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.518159 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.518358 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.518489 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":94,"threads_started":1}
I20260927 00:44:27.518728 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.513845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50917 (local address 127.0.98.3:40721)
0927 00:44:27.514126 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.514135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.514171 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:44:27.514362 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.514366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.514424 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.514535 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:27.515300 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.516074 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.517418 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.518149 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.518260 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.518346 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.518496 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.518572 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
I20260927 00:44:27.518631 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.513662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.98.1:50917)
0927 00:44:27.514104 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:27.514120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:27.514197 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:27.514558 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:27.514561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:27.514597 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:27.515174 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.515181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.516185 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.516189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.517284 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.517308 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.518258 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.518264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.518373 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.518391 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:44:27.518458 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:44:27.518511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":78,"spinlock_wait_cycles":31616,"thread_start_us":158,"threads_started":1}
I20260927 00:44:27.519387 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723cd42e43ba4675bcfc0eccced43fb1" candidate_uuid: "72447db69b78416681da6082ddfad3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371" is_pre_election: true
I20260927 00:44:27.519464 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723cd42e43ba4675bcfc0eccced43fb1" candidate_uuid: "72447db69b78416681da6082ddfad3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184" is_pre_election: true
I20260927 00:44:27.519586 2419 raft_consensus.cc:2454] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72447db69b78416681da6082ddfad3f8 in term 0.
I20260927 00:44:27.519646 2559 raft_consensus.cc:2454] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72447db69b78416681da6082ddfad3f8 in term 0.
I20260927 00:44:27.520140 2211 leader_election.cc:302] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, a69bf07465e943feb0c496bd8f0d6184, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:27.520435 2727 raft_consensus.cc:2781] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:27.520496 2727 raft_consensus.cc:482] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:27.520533 2727 raft_consensus.cc:3037] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.521659 2727 raft_consensus.cc:504] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.522034 2727 leader_election.cc:288] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [CANDIDATE]: Term 1 election: Requested vote from peers ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963), a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721)
I20260927 00:44:27.522358 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723cd42e43ba4675bcfc0eccced43fb1" candidate_uuid: "72447db69b78416681da6082ddfad3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371"
I20260927 00:44:27.522418 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723cd42e43ba4675bcfc0eccced43fb1" candidate_uuid: "72447db69b78416681da6082ddfad3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69bf07465e943feb0c496bd8f0d6184"
I20260927 00:44:27.522505 2419 raft_consensus.cc:3037] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.522562 2559 raft_consensus.cc:3037] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.523442 2559 raft_consensus.cc:2454] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72447db69b78416681da6082ddfad3f8 in term 1.
I20260927 00:44:27.523574 2419 raft_consensus.cc:2454] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72447db69b78416681da6082ddfad3f8 in term 1.
I20260927 00:44:27.523739 2213 leader_election.cc:302] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72447db69b78416681da6082ddfad3f8, a69bf07465e943feb0c496bd8f0d6184; no voters:
I20260927 00:44:27.523940 2727 raft_consensus.cc:2781] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:27.524155 2727 raft_consensus.cc:686] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Running, Role: LEADER
I20260927 00:44:27.524438 2727 consensus_queue.cc:237] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.526872 2119 catalog_manager.cc:5495] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72447db69b78416681da6082ddfad3f8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "72447db69b78416681da6082ddfad3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:27.549881 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.545306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.0.98.1:38881)
0927 00:44:27.545620 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.545628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.545639 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:27.545714 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.545717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.545835 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.545985 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:27.546267 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.547275 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.548233 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.548495 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.548618 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.548924 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.549398 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.549542 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.549611 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.549666 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260927 00:44:27.560261 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.556675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.98.1:51171)
0927 00:44:27.557139 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:27.557156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:27.557282 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:27.557516 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:27.557519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:27.557527 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:27.557665 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.557670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.558423 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.558427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.559056 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.559064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.559516 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.559520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.559616 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.559672 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.559978 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.559984 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:44:27.560045 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:44:27.560112 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":125,"threads_started":1}
I20260927 00:44:27.560250 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.556819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51171 (local address 127.0.98.3:40721)
0927 00:44:27.557063 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.557068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.557246 (+ 178us) server_negotiation.cc:407] Connection header received
0927 00:44:27.557369 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.557373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.557424 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.557506 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:27.557797 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.558307 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.559174 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.559407 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.559515 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.559816 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.559882 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.559960 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.560055 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.560103 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 00:44:27.566051 2559 raft_consensus.cc:1261] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Refusing update from remote peer 72447db69b78416681da6082ddfad3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:27.566078 2419 raft_consensus.cc:1261] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Refusing update from remote peer 72447db69b78416681da6082ddfad3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:27.566495 2727 consensus_queue.cc:1035] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:27.566725 2732 consensus_queue.cc:1035] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:27.583652 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.580483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56505 (local address 127.0.98.1:38881)
0927 00:44:27.580580 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.580585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.580711 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:44:27.580817 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.580820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.580855 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.580911 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:27.581060 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.581539 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.582642 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.582878 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.582984 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.583222 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.583304 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.583379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.583469 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.583519 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:44:27.583653 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.580408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.98.3:56505)
0927 00:44:27.580640 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:27.580657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:27.580746 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:27.580914 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:27.580916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:27.580922 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:27.581002 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.581006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.581633 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.581636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.582519 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.582527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.582982 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.582985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.583073 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.583107 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.583381 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.583390 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:44:27.583452 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:44:27.583528 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260927 00:44:27.605830 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:27.606010 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:27.608191 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Sending DeleteTablet for 3 replicas of tablet 723cd42e43ba4675bcfc0eccced43fb1
I20260927 00:44:27.608795 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32850
I20260927 00:44:27.608938 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57872
I20260927 00:44:27.608983 2710 tablet_replica.cc:330] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: stopping tablet replica
I20260927 00:44:27.609107 2710 raft_consensus.cc:2229] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:27.609138 2712 tablet_replica.cc:330] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: stopping tablet replica
I20260927 00:44:27.609264 2712 raft_consensus.cc:2229] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:27.609382 2712 raft_consensus.cc:2258] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.609426 2710 raft_consensus.cc:2258] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.609421 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 723cd42e43ba4675bcfc0eccced43fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260927 00:44:27.609586 2711 tablet_replica.cc:330] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: stopping tablet replica
I20260927 00:44:27.609685 2711 raft_consensus.cc:2229] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:27.609823 2711 raft_consensus.cc:2258] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:27.610365 2710 ts_tablet_manager.cc:1905] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.610455 2712 ts_tablet_manager.cc:1905] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.610589 2711 ts_tablet_manager.cc:1905] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:27.613147 2711 ts_tablet_manager.cc:1918] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:27.613224 2711 log.cc:1196] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/723cd42e43ba4675bcfc0eccced43fb1
I20260927 00:44:27.612859 2710 ts_tablet_manager.cc:1918] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:27.613349 2712 ts_tablet_manager.cc:1918] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:27.613437 2712 log.cc:1196] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/723cd42e43ba4675bcfc0eccced43fb1
I20260927 00:44:27.613492 2711 ts_tablet_manager.cc:1939] T 723cd42e43ba4675bcfc0eccced43fb1 P a69bf07465e943feb0c496bd8f0d6184: Deleting consensus metadata
I20260927 00:44:27.613507 2710 log.cc:1196] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/723cd42e43ba4675bcfc0eccced43fb1
I20260927 00:44:27.613719 2712 ts_tablet_manager.cc:1939] T 723cd42e43ba4675bcfc0eccced43fb1 P ab7fc76f58cd46fb9325cafee0b16371: Deleting consensus metadata
I20260927 00:44:27.613921 2710 ts_tablet_manager.cc:1939] T 723cd42e43ba4675bcfc0eccced43fb1 P 72447db69b78416681da6082ddfad3f8: Deleting consensus metadata
I20260927 00:44:27.614223 2109 catalog_manager.cc:4841] TS a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3:40721): tablet 723cd42e43ba4675bcfc0eccced43fb1 (table test-table [id=c03b18a7f8d44e098d93668f5306ea33]) successfully deleted
I20260927 00:44:27.614588 2107 catalog_manager.cc:4841] TS 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881): tablet 723cd42e43ba4675bcfc0eccced43fb1 (table test-table [id=c03b18a7f8d44e098d93668f5306ea33]) successfully deleted
I20260927 00:44:27.614737 2107 catalog_manager.cc:4841] TS ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963): tablet 723cd42e43ba4675bcfc0eccced43fb1 (table test-table [id=c03b18a7f8d44e098d93668f5306ea33]) successfully deleted
I20260927 00:44:27.614856 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:27.615279 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:27.619757 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 27dba8f5159b4b31ad12087df25293a8 (DEFAULT_TABLE table=test-table [id=97d093aaf0ef45a98e5ede7ff8f9dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.619906 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 27dba8f5159b4b31ad12087df25293a8 (DEFAULT_TABLE table=test-table [id=97d093aaf0ef45a98e5ede7ff8f9dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.620014 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dba8f5159b4b31ad12087df25293a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.620136 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dba8f5159b4b31ad12087df25293a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.622123 2722 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: Bootstrap starting.
I20260927 00:44:27.622417 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 27dba8f5159b4b31ad12087df25293a8 (DEFAULT_TABLE table=test-table [id=97d093aaf0ef45a98e5ede7ff8f9dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:27.622623 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27dba8f5159b4b31ad12087df25293a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:27.622946 2722 tablet_bootstrap.cc:654] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.623628 2721 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: Bootstrap starting.
I20260927 00:44:27.624461 2721 tablet_bootstrap.cc:654] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.624620 2723 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8: Bootstrap starting.
I20260927 00:44:27.625555 2723 tablet_bootstrap.cc:654] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:27.625727 2721 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: No bootstrap required, opened a new log
I20260927 00:44:27.625830 2721 ts_tablet_manager.cc:1397] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:27.626236 2721 raft_consensus.cc:348] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.626344 2721 raft_consensus.cc:374] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.626374 2721 raft_consensus.cc:729] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.626477 2721 consensus_queue.cc:260] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.626754 2721 ts_tablet_manager.cc:1428] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:27.627421 2723 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8: No bootstrap required, opened a new log
I20260927 00:44:27.627492 2723 ts_tablet_manager.cc:1397] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:27.627872 2723 raft_consensus.cc:348] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.627974 2723 raft_consensus.cc:374] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.628000 2723 raft_consensus.cc:729] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72447db69b78416681da6082ddfad3f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.628098 2723 consensus_queue.cc:260] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.628307 2723 ts_tablet_manager.cc:1428] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:27.630518 2722 tablet_bootstrap.cc:492] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: No bootstrap required, opened a new log
I20260927 00:44:27.630609 2722 ts_tablet_manager.cc:1397] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260927 00:44:27.630972 2722 raft_consensus.cc:348] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.631081 2722 raft_consensus.cc:374] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:27.631120 2722 raft_consensus.cc:729] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fc76f58cd46fb9325cafee0b16371, State: Initialized, Role: FOLLOWER
I20260927 00:44:27.631244 2722 consensus_queue.cc:260] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.631502 2722 ts_tablet_manager.cc:1428] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:27.881888 2695 raft_consensus.cc:482] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:27.882030 2695 raft_consensus.cc:504] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.882515 2695 leader_election.cc:288] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963)
I20260927 00:44:27.883107 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dba8f5159b4b31ad12087df25293a8" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371" is_pre_election: true
I20260927 00:44:27.883108 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dba8f5159b4b31ad12087df25293a8" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8" is_pre_election: true
I20260927 00:44:27.883266 2279 raft_consensus.cc:2454] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 0.
I20260927 00:44:27.883267 2419 raft_consensus.cc:2454] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 0.
I20260927 00:44:27.883620 2491 leader_election.cc:302] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69bf07465e943feb0c496bd8f0d6184, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:27.883857 2695 raft_consensus.cc:2781] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:27.883929 2695 raft_consensus.cc:482] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:27.883977 2695 raft_consensus.cc:3037] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.885056 2695 raft_consensus.cc:504] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.885422 2695 leader_election.cc:288] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 election: Requested vote from peers 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881), ab7fc76f58cd46fb9325cafee0b16371 (127.0.98.2:38963)
I20260927 00:44:27.885895 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dba8f5159b4b31ad12087df25293a8" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72447db69b78416681da6082ddfad3f8"
I20260927 00:44:27.885898 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27dba8f5159b4b31ad12087df25293a8" candidate_uuid: "a69bf07465e943feb0c496bd8f0d6184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fc76f58cd46fb9325cafee0b16371"
I20260927 00:44:27.886014 2279 raft_consensus.cc:3037] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.886042 2419 raft_consensus.cc:3037] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:27.887125 2279 raft_consensus.cc:2454] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 1.
I20260927 00:44:27.887140 2419 raft_consensus.cc:2454] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69bf07465e943feb0c496bd8f0d6184 in term 1.
I20260927 00:44:27.887459 2491 leader_election.cc:302] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69bf07465e943feb0c496bd8f0d6184, ab7fc76f58cd46fb9325cafee0b16371; no voters:
I20260927 00:44:27.887642 2695 raft_consensus.cc:2781] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:27.887756 2695 raft_consensus.cc:686] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 1 LEADER]: Becoming Leader. State: Replica: a69bf07465e943feb0c496bd8f0d6184, State: Running, Role: LEADER
I20260927 00:44:27.887917 2695 consensus_queue.cc:237] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } }
I20260927 00:44:27.889688 2122 catalog_manager.cc:5495] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69bf07465e943feb0c496bd8f0d6184 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a69bf07465e943feb0c496bd8f0d6184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69bf07465e943feb0c496bd8f0d6184" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40721 } health_report { overall_health: HEALTHY } } }
I20260927 00:44:27.927389 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.924666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.0.98.3:40721)
0927 00:44:27.924823 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.924827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.924840 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:27.924901 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.924903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.924944 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.925012 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:27.925225 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.925711 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.926496 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.926710 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.926791 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.927002 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.927065 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.927147 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.927243 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.927284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:44:27.935833 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.933012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59901 (local address 127.0.98.3:40721)
0927 00:44:27.933109 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:27.933116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:27.933129 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:27.933167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:27.933170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:27.933224 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:27.933303 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:27.933524 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.934047 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.934839 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.935041 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.935146 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.935409 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.935475 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.935553 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:27.935635 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 00:44:27.935693 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":41}
I20260927 00:44:27.935833 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:27.932828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40721 (local address 127.0.98.3:59901)
0927 00:44:27.932965 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:27.932979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:27.933079 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:27.933312 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:27.933314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:27.933320 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:27.933423 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.933428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.934155 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.934161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.934724 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:27.934731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:27.935144 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:27.935147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:27.935269 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:27.935300 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:27.935557 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:27.935564 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:44:27.935628 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:44:27.935732 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":83}
I20260927 00:44:27.941493 2419 raft_consensus.cc:1261] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Refusing update from remote peer a69bf07465e943feb0c496bd8f0d6184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:27.941535 2279 raft_consensus.cc:1261] T 27dba8f5159b4b31ad12087df25293a8 P 72447db69b78416681da6082ddfad3f8 [term 1 FOLLOWER]: Refusing update from remote peer a69bf07465e943feb0c496bd8f0d6184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:27.942010 2651 consensus_queue.cc:1035] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7fc76f58cd46fb9325cafee0b16371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:27.942137 2690 consensus_queue.cc:1035] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72447db69b78416681da6082ddfad3f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:44:27.955237 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:44:29.507750 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:29.508152 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:29.511535 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36278c05cd6d4e159370c7bffa47069b: Sending DeleteTablet for 3 replicas of tablet 27dba8f5159b4b31ad12087df25293a8
I20260927 00:44:29.512085 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dba8f5159b4b31ad12087df25293a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32850
I20260927 00:44:29.512140 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dba8f5159b4b31ad12087df25293a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57872
I20260927 00:44:29.512488 2773 tablet_replica.cc:330] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: stopping tablet replica
I20260927 00:44:29.512639 2773 raft_consensus.cc:2229] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:29.512779 2773 raft_consensus.cc:2258] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:29.512789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2194
I20260927 00:44:29.513909 2773 ts_tablet_manager.cc:1905] T 27dba8f5159b4b31ad12087df25293a8 P ab7fc76f58cd46fb9325cafee0b16371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:29.517299 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 27dba8f5159b4b31ad12087df25293a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260927 00:44:29.517750 2774 tablet_replica.cc:330] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: stopping tablet replica
I20260927 00:44:29.517944 2774 raft_consensus.cc:2229] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:29.518234 2774 raft_consensus.cc:2258] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:29.518790 2774 ts_tablet_manager.cc:1905] T 27dba8f5159b4b31ad12087df25293a8 P a69bf07465e943feb0c496bd8f0d6184: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:44:29.523327 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38881 (error 108)
I20260927 00:44:29.523399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2334
I20260927 00:44:29.525094 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:29.524610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.0.1:48064)
0927 00:44:29.524871 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:29.524963 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
W20260927 00:44:29.525243 2107 catalog_manager.cc:4584] TS 72447db69b78416681da6082ddfad3f8 (127.0.98.1:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27dba8f5159b4b31ad12087df25293a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38881: connect: Connection refused (error 111)
I20260927 00:44:29.568939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2474
I20260927 00:44:29.570087 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:29.569049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38963 (local address 127.0.0.1:36150)
0927 00:44:29.569908 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:29.569975 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":746}
I20260927 00:44:29.579118 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:29.578744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38881 (local address 127.0.0.1:48076)
0927 00:44:29.578971 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:29.579025 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 00:44:29.579281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2090
2026-09-27T00:44:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7774 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:29 dist-test-slave-l5sb krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:29 dist-test-slave-l5sb krb5kdc[2779](info): set up 2 sockets
Sep 27 00:44:29 dist-test-slave-l5sb krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260927 00:44:30.629702 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:30 dist-test-slave-l5sb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:30.752739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:30.863874 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:30.864147 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:30.864189 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:30.868147 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:30.868201 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:30.868219 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:30.868242 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:30.868257 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:30.873308 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:30.874274 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:30.875188 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:44:30.883886 2792 server_base.cc:1034] running on GCE node
W20260927 00:44:30.883841 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:30.883795 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:30.883827 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:30.884531 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:30.885573 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:30.886722 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790469870886710 us; error 57 us; skew 500 ppm
Sep 27 00:44:30 dist-test-slave-l5sb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:30.889394 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:30.890324 2792 webserver.cc:460] Webserver started at http://127.0.98.62:33511/ using document root <none> and password file <none>
I20260927 00:44:30.890870 2792 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:30.890956 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:30.891177 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:30.893163 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "89794f6470a545ec8e083b6385f9d084"
format_stamp: "Formatted at 2026-09-27 00:44:30 on dist-test-slave-l5sb"
server_key: "45eb13c647728bf89fc42a39d8ae01fa"
server_key_iv: "8599a2aefe0b93d6523557a19f8e5975"
server_key_version: "encryptionkey@0"
I20260927 00:44:30.893721 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "89794f6470a545ec8e083b6385f9d084"
format_stamp: "Formatted at 2026-09-27 00:44:30 on dist-test-slave-l5sb"
server_key: "45eb13c647728bf89fc42a39d8ae01fa"
server_key_iv: "8599a2aefe0b93d6523557a19f8e5975"
server_key_version: "encryptionkey@0"
I20260927 00:44:30.897182 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:44:30.899613 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:30.900830 2792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:30.900978 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "89794f6470a545ec8e083b6385f9d084"
format_stamp: "Formatted at 2026-09-27 00:44:30 on dist-test-slave-l5sb"
server_key: "45eb13c647728bf89fc42a39d8ae01fa"
server_key_iv: "8599a2aefe0b93d6523557a19f8e5975"
server_key_version: "encryptionkey@0"
I20260927 00:44:30.901087 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:30.917665 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:30.920531 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:30.920734 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:30.927712 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33221
I20260927 00:44:30.927744 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33221 every 8 connection(s)
I20260927 00:44:30.928628 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:30.931260 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:30.936442 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: Bootstrap starting.
I20260927 00:44:30.938424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2792
I20260927 00:44:30.938539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:30.938817 392 external_mini_cluster.cc:1428] Setting key 6fc139ec6d58a1d2b5ee0013f2842bd0
I20260927 00:44:30.938954 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:30.939754 2860 log.cc:824] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:30.941929 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: No bootstrap required, opened a new log
I20260927 00:44:30.946157 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } }
I20260927 00:44:30.946363 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:30.946429 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89794f6470a545ec8e083b6385f9d084, State: Initialized, Role: FOLLOWER
I20260927 00:44:30.946838 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } }
I20260927 00:44:30.946991 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:30.947077 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:30.947211 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:44:30 dist-test-slave-l5sb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:30.948266 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } }
I20260927 00:44:30.948616 2860 leader_election.cc:302] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89794f6470a545ec8e083b6385f9d084; no voters:
I20260927 00:44:30.948985 2860 leader_election.cc:288] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:30.949087 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:30.949339 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [term 1 LEADER]: Becoming Leader. State: Replica: 89794f6470a545ec8e083b6385f9d084, State: Running, Role: LEADER
I20260927 00:44:30.949738 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } }
I20260927 00:44:30.950248 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:30.951411 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89794f6470a545ec8e083b6385f9d084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } } }
I20260927 00:44:30.951562 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:30.951932 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89794f6470a545ec8e083b6385f9d084. Latest consensus state: current_term: 1 leader_uuid: "89794f6470a545ec8e083b6385f9d084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89794f6470a545ec8e083b6385f9d084" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33221 } } }
I20260927 00:44:30.952085 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:30.954320 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:30.939968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.0.98.62:33221)
0927 00:44:30.940430 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:30.940444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:30.940476 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:44:30.941304 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:30.941327 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:30.941712 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:30.942090 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:30.942783 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:30.944262 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:30.945122 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:30.945847 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:30.945998 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:30.948921 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:30.948926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:30.948938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:30.948987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:30.951253 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:30.951802 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:30.951810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:30.951819 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:30.951970 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:30.952304 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:30.952310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:30.952315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:30.952846 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:30.953001 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:30.953280 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:44:30.953558 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":99,"threads_started":1}
W20260927 00:44:30.955585 2880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:30.955662 2880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:30.955739 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:30.956372 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:30.961370 2874 catalog_manager.cc:1347] Generated new cluster ID: 4e28eedf511944ceae7897c264758a02
I20260927 00:44:30.961433 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:30.968427 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:30.969614 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:30.981025 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89794f6470a545ec8e083b6385f9d084: Generated new TSK 0
I20260927 00:44:30.982012 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:31.016662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:31.130450 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:31.130699 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:31.130740 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:31.134871 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:31.134927 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:31.135010 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:31.140129 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:31.141139 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:31.142231 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:31.151875 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:31.151890 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:31.152180 2886 server_base.cc:1034] running on GCE node
W20260927 00:44:31.151901 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:31.152650 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:31.153327 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:31.154513 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790469871154482 us; error 48 us; skew 500 ppm
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:31.157166 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:31.158144 2886 webserver.cc:460] Webserver started at http://127.0.98.1:45485/ using document root <none> and password file <none>
I20260927 00:44:31.158703 2886 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:31.158766 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:31.158927 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:31.160888 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "9bb0671d258341ef8960c45dd22e9993"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "2323e1ff2cceb7fa81a7cc478e382d42"
server_key_iv: "14301b4a946deb249eb86dd23aff510d"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.161412 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9bb0671d258341ef8960c45dd22e9993"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "2323e1ff2cceb7fa81a7cc478e382d42"
server_key_iv: "14301b4a946deb249eb86dd23aff510d"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.165120 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:31.167659 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.168960 2886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:31.169106 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "9bb0671d258341ef8960c45dd22e9993"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "2323e1ff2cceb7fa81a7cc478e382d42"
server_key_iv: "14301b4a946deb249eb86dd23aff510d"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.169224 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:31.185990 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:31.189043 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:31.189268 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:31.189901 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:31.190758 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:31.190809 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.190881 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:31.190917 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.200738 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42043
I20260927 00:44:31.200783 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42043 every 8 connection(s)
I20260927 00:44:31.201751 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:31.202885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 2886
I20260927 00:44:31.202983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:31.203289 392 external_mini_cluster.cc:1428] Setting key 0909cbd506e49dd0ab8de66da4120768
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:31.215803 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.203800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35499 (local address 127.0.98.62:33221)
0927 00:44:31.203962 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:31.203966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:31.204847 (+ 881us) server_negotiation.cc:407] Connection header received
0927 00:44:31.206427 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:31.206431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:31.206484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:31.206564 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:31.207735 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.208264 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.209473 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.209710 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.209866 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.212229 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:31.212235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:31.212243 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:31.212268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:31.213683 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.214205 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.214210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.214212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.214298 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.214634 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.214638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.214640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.214830 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:31.214939 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:31.215524 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 00:44:31.215645 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:44:31.216862 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.204230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33221 (local address 127.0.98.1:35499)
0927 00:44:31.204697 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:31.204751 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:31.206210 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:31.206744 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:31.206752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:31.207099 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:31.207574 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.207587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.208391 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.208395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.209349 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.209358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.209886 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.209890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.210078 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.210680 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:31.210696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:31.212067 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:31.213826 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.213833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.213847 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.214100 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.214398 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.214401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.214403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.214538 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.214935 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:31.214941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:31.215245 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 00:44:31.215462 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 00:44:31.215830 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260927 00:44:31.218187 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33221
I20260927 00:44:31.218446 3015 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:31.218945 3015 heartbeater.cc:511] Master 127.0.98.62:33221 requested a full tablet report, sending...
I20260927 00:44:31.220492 2824 ts_manager.cc:194] Registered new tserver with Master: 9bb0671d258341ef8960c45dd22e9993 (127.0.98.1:42043)
I20260927 00:44:31.221715 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:31.233027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:31.346484 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:31.346747 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:31.346791 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:31.351130 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:31.351188 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:31.351248 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:31.356395 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:31.357334 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:31.358379 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:31.367764 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:31.367797 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:31.367764 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:31.367964 3020 server_base.cc:1034] running on GCE node
I20260927 00:44:31.368350 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:31.368979 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:31.370158 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790469871370146 us; error 26 us; skew 500 ppm
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:31.372877 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:31.373801 3020 webserver.cc:460] Webserver started at http://127.0.98.2:37857/ using document root <none> and password file <none>
I20260927 00:44:31.374341 3020 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:31.374430 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:31.374639 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:31.376540 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "0143ae8ac07141619c7b7722bfcc9005"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "1f5d639958ad27bdb1919e2b190ac733"
server_key_iv: "a6131723ae0691f9c2e05c4fcb668247"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.377087 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0143ae8ac07141619c7b7722bfcc9005"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "1f5d639958ad27bdb1919e2b190ac733"
server_key_iv: "a6131723ae0691f9c2e05c4fcb668247"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.380519 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:31.382779 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.384032 3020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:31.384188 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "0143ae8ac07141619c7b7722bfcc9005"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "1f5d639958ad27bdb1919e2b190ac733"
server_key_iv: "a6131723ae0691f9c2e05c4fcb668247"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.384296 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:31.393857 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:31.396963 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:31.397162 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:31.397701 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:31.398483 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:31.398562 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.398633 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:31.398669 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.407671 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46135
I20260927 00:44:31.407691 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46135 every 8 connection(s)
I20260927 00:44:31.408558 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:31.409123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3020
I20260927 00:44:31.409231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:31.409507 392 external_mini_cluster.cc:1428] Setting key 357749b372870d979bbbb4013320ed19
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:31.426697 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.410403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42241 (local address 127.0.98.62:33221)
0927 00:44:31.410547 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:31.410552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:31.411607 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 00:44:31.413359 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:31.413362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:31.413407 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:31.413477 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:31.415167 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.415713 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.417557 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.417761 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.417923 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.422900 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:31.422905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:31.422907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:31.422936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:31.424409 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.425023 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.425025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.425027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.425098 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.425493 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.425495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.425496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.425704 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:31.425842 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:31.426475 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 00:44:31.426576 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:44:31.428020 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.410936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33221 (local address 127.0.98.2:42241)
0927 00:44:31.411431 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:31.411482 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:31.413115 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:31.413676 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:31.413688 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:31.414252 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:31.414934 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.414958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.415856 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.415863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.417435 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.417449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.417965 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.417969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.418134 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.418747 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:31.418763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:31.422694 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:31.424518 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.424528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.424546 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.424921 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.425184 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.425198 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.425203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.425397 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.425844 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:31.425854 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:31.426215 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 00:44:31.426444 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 00:44:31.426839 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20260927 00:44:31.429301 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33221
I20260927 00:44:31.429549 3149 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:31.430094 3149 heartbeater.cc:511] Master 127.0.98.62:33221 requested a full tablet report, sending...
I20260927 00:44:31.431154 2824 ts_manager.cc:194] Registered new tserver with Master: 0143ae8ac07141619c7b7722bfcc9005 (127.0.98.2:46135)
I20260927 00:44:31.431733 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:31.438722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:31.546506 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:31.546757 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:31.546800 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:31.551100 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:31.551170 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:31.551290 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:31.556481 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:31.557436 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:31.558455 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:31.567832 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:31.567862 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:31.567881 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:31.568213 3154 server_base.cc:1034] running on GCE node
I20260927 00:44:31.568536 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:31.569126 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:31.570307 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790469871570274 us; error 44 us; skew 500 ppm
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:31.572919 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:31.573817 3154 webserver.cc:460] Webserver started at http://127.0.98.3:40691/ using document root <none> and password file <none>
I20260927 00:44:31.574373 3154 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:31.574461 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:31.574684 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:31.576629 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "78ae19a09d6648ce8567795ce24a180f"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "0d0ae01b9f60e2040b5ccf9ba3c1aa65"
server_key_iv: "1dc237dd51b5a11d8e2077e7874fbb58"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.577186 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78ae19a09d6648ce8567795ce24a180f"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "0d0ae01b9f60e2040b5ccf9ba3c1aa65"
server_key_iv: "1dc237dd51b5a11d8e2077e7874fbb58"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.580720 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:31.583117 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.584203 3154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:31.584343 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "78ae19a09d6648ce8567795ce24a180f"
format_stamp: "Formatted at 2026-09-27 00:44:31 on dist-test-slave-l5sb"
server_key: "0d0ae01b9f60e2040b5ccf9ba3c1aa65"
server_key_iv: "1dc237dd51b5a11d8e2077e7874fbb58"
server_key_version: "encryptionkey@0"
I20260927 00:44:31.584455 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:31.606637 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:31.609504 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:31.609704 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:31.610303 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:31.611085 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:31.611157 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.611229 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:31.611264 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:31.619781 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34547
I20260927 00:44:31.619805 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34547 every 8 connection(s)
I20260927 00:44:31.620600 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:31.624697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3154
I20260927 00:44:31.624830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:31.625082 392 external_mini_cluster.cc:1428] Setting key 2720ca31b54ac82e2176e5b189eb804f
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:31.633461 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.622315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50561 (local address 127.0.98.62:33221)
0927 00:44:31.622450 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:31.622455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:31.623301 (+ 846us) server_negotiation.cc:407] Connection header received
0927 00:44:31.624607 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:31.624611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:31.624669 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:31.624781 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:31.626160 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.626817 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.627993 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.628206 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.628289 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.630621 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:31.630624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:31.630626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:31.630650 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:31.631868 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.632284 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.632286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.632287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.632355 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:31.632618 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:31.632620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:31.632621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:31.632787 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:31.632859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:31.633261 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 00:44:31.633346 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:44:31.634295 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:31.622744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33221 (local address 127.0.98.3:50561)
0927 00:44:31.623153 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:31.623194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:31.624399 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:31.624949 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:31.624962 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:31.625354 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:31.626018 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.626039 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.626933 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.626936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.627884 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:31.627893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:31.628329 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:31.628331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:31.628459 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:31.629056 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:31.629071 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:31.630479 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:31.631962 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.631966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.631977 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.632205 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.632427 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:31.632429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:31.632431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:31.632546 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:31.632868 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:31.632873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:31.633074 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 00:44:31.633221 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 00:44:31.633440 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260927 00:44:31.635367 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33221
I20260927 00:44:31.635555 3283 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:31.635947 3283 heartbeater.cc:511] Master 127.0.98.62:33221 requested a full tablet report, sending...
I20260927 00:44:31.636859 2824 ts_manager.cc:194] Registered new tserver with Master: 78ae19a09d6648ce8567795ce24a180f (127.0.98.3:34547)
I20260927 00:44:31.637362 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50561
I20260927 00:44:31.638885 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:44:31.651399 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:31.649827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.0.98.62:33221)
0927 00:44:31.649986 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:31.649991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:31.650156 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:44:31.650340 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:31.650346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:31.650405 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:31.650495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:31.651239 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44846: BlockingRecv error: recv got EOF from 127.0.0.1:44846 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:31.652644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2886
I20260927 00:44:31.658198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3020
I20260927 00:44:31.663043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3154
I20260927 00:44:31.668051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 2792
2026-09-27T00:44:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2086 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[3295](info): set up 2 sockets
Sep 27 00:44:31 dist-test-slave-l5sb krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260927 00:44:32.705720 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:32 dist-test-slave-l5sb krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:32.829005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:32.935884 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:32.936133 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:32.936173 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:32.940222 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:32.940277 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:32.940296 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:32.940313 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:32.940366 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:32.945422 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:32.946502 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:32.947518 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:32.955677 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:32.955709 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:32.955750 3308 server_base.cc:1034] running on GCE node
W20260927 00:44:32.955700 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:32.956404 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:32.957511 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:32.958698 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790469872958666 us; error 52 us; skew 500 ppm
Sep 27 00:44:32 dist-test-slave-l5sb krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:32.961289 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:32.962159 3308 webserver.cc:460] Webserver started at http://127.0.98.62:45035/ using document root <none> and password file <none>
I20260927 00:44:32.962692 3308 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:32.962780 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:32.962996 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:32.964949 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "9021823a7d9a407ab75882a9c2d69f36"
format_stamp: "Formatted at 2026-09-27 00:44:32 on dist-test-slave-l5sb"
server_key: "22dc3e5a3e5619a134b96c3e0c033086"
server_key_iv: "52bfc483943ab5e51a5fdc7868315a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:32.965505 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "9021823a7d9a407ab75882a9c2d69f36"
format_stamp: "Formatted at 2026-09-27 00:44:32 on dist-test-slave-l5sb"
server_key: "22dc3e5a3e5619a134b96c3e0c033086"
server_key_iv: "52bfc483943ab5e51a5fdc7868315a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:32.969118 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:32.971486 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:32.972618 3308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:32.972757 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "9021823a7d9a407ab75882a9c2d69f36"
format_stamp: "Formatted at 2026-09-27 00:44:32 on dist-test-slave-l5sb"
server_key: "22dc3e5a3e5619a134b96c3e0c033086"
server_key_iv: "52bfc483943ab5e51a5fdc7868315a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:32.972863 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:32.986768 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:32.989483 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:32.989679 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:32.996273 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40243
I20260927 00:44:32.996567 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40243 every 8 connection(s)
I20260927 00:44:32.997212 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:32.999939 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:33.004401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3308
I20260927 00:44:33.004525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:33.004815 392 external_mini_cluster.cc:1428] Setting key 08f61470147c338b1e93461426291aac
I20260927 00:44:33.005613 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: Bootstrap starting.
I20260927 00:44:33.007952 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:33.008879 3376 log.cc:824] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:33.010993 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: No bootstrap required, opened a new log
Sep 27 00:44:33 dist-test-slave-l5sb krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:33.015240 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } }
I20260927 00:44:33.015511 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:33.015578 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9021823a7d9a407ab75882a9c2d69f36, State: Initialized, Role: FOLLOWER
I20260927 00:44:33.016059 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } }
I20260927 00:44:33.016184 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:33.016278 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:33.016377 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:33.017218 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } }
I20260927 00:44:33.017544 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9021823a7d9a407ab75882a9c2d69f36; no voters:
I20260927 00:44:33.017896 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:33.017963 3381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:33.018155 3381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [term 1 LEADER]: Becoming Leader. State: Replica: 9021823a7d9a407ab75882a9c2d69f36, State: Running, Role: LEADER
I20260927 00:44:33.018494 3381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } }
I20260927 00:44:33.019131 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:33.019546 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:33.006141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.98.62:40243)
0927 00:44:33.006539 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:33.006554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:33.006588 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:44:33.007199 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:33.007216 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:33.007532 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:33.007805 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:33.008743 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:33.010005 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:33.011132 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:33.011597 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:33.011720 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:33.014518 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:33.014524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:33.014536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:33.014560 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:33.016401 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:33.016842 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:33.016848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:33.016853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:33.016978 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:33.017249 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:33.017254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:33.017256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:33.017689 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:33.017904 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:33.018228 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:44:33.018642 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260927 00:44:33.019655 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9021823a7d9a407ab75882a9c2d69f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } } }
I20260927 00:44:33.019951 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:33.020215 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9021823a7d9a407ab75882a9c2d69f36. Latest consensus state: current_term: 1 leader_uuid: "9021823a7d9a407ab75882a9c2d69f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9021823a7d9a407ab75882a9c2d69f36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40243 } } }
I20260927 00:44:33.020371 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:33.022821 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:44:33.024139 3397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:33.024212 3397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:33.024268 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:33.025130 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:33.030547 3390 catalog_manager.cc:1347] Generated new cluster ID: 7f7f6890d904478f808da5bb136341ec
I20260927 00:44:33.030606 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:33.048837 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:33.043280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.0.98.62:40243)
0927 00:44:33.043386 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:33.043391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:33.043650 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:44:33.043784 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:33.043788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:33.043842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:33.043931 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:33.044471 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:33.045287 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:33.046423 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:33.046783 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:33.046886 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:33.047785 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:33.047789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:33.047792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:33.048389 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:33.048422 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:33.048441 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:33.048530 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:33.048628 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 00:44:33.048690 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:44:33.052848 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:33.053733 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:33.060026 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9021823a7d9a407ab75882a9c2d69f36: Generated new TSK 0
I20260927 00:44:33.060766 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:44:33.065975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3308
2026-09-27T00:44:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1395 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:33 dist-test-slave-l5sb krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:33 dist-test-slave-l5sb krb5kdc[3409](info): set up 2 sockets
Sep 27 00:44:33 dist-test-slave-l5sb krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260927 00:44:34.153986 392 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:34 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:34.276964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:34.388759 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:34.389063 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:34.389140 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:34.393106 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:34.393195 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:34.393236 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:34.393294 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:34.393334 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:34.398488 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:34.399556 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:34.400559 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:34.408896 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:34.408896 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:34.408943 3422 server_base.cc:1034] running on GCE node
W20260927 00:44:34.408908 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:34.409729 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:34.410792 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:34.411973 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790469874411947 us; error 39 us; skew 500 ppm
Sep 27 00:44:34 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:34.414686 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:34.415503 3422 webserver.cc:460] Webserver started at http://127.0.98.62:36017/ using document root <none> and password file <none>
I20260927 00:44:34.416042 3422 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:34.416127 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:34.416347 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:34.418387 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "50ea834c8f8141aeaeabfd81578a84bb"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "262499c265876aff2ef0a80762628d20"
server_key_iv: "c6f8d0748cfe1b5322c58b12ecaeda71"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.418932 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "50ea834c8f8141aeaeabfd81578a84bb"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "262499c265876aff2ef0a80762628d20"
server_key_iv: "c6f8d0748cfe1b5322c58b12ecaeda71"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.422698 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:44:34.425022 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:34.426313 3422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:34.426448 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "50ea834c8f8141aeaeabfd81578a84bb"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "262499c265876aff2ef0a80762628d20"
server_key_iv: "c6f8d0748cfe1b5322c58b12ecaeda71"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.426558 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:34.450446 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:34.453364 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:34.453563 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:34.460433 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43803
I20260927 00:44:34.460736 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43803 every 8 connection(s)
I20260927 00:44:34.461366 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:34.462605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3422
I20260927 00:44:34.462706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:34.462999 392 external_mini_cluster.cc:1428] Setting key 0c0eb3e84fad40d504da822d4848a70a
I20260927 00:44:34.464393 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:34.471240 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb: Bootstrap starting.
Sep 27 00:44:34 dist-test-slave-l5sb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:34.473835 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:34.474767 3490 log.cc:824] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:34.476894 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb: No bootstrap required, opened a new log
I20260927 00:44:34.478288 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:34.464338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.0.98.62:43803)
0927 00:44:34.464768 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:34.464786 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:34.464827 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:44:34.465620 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:34.465652 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:34.466075 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:34.466444 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:34.467439 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.468933 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.469819 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.470507 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.470666 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:34.473281 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:34.473287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:34.473301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:34.473330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:34.475289 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:34.475723 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:34.475728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:34.475733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:34.475858 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:34.476105 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:34.476108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:34.476109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:34.476573 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:34.476756 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:34.476945 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:44:34.477381 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260927 00:44:34.480901 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 00:44:34.481123 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:34.481168 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea834c8f8141aeaeabfd81578a84bb, State: Initialized, Role: FOLLOWER
I20260927 00:44:34.481559 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 00:44:34.481674 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:34.481719 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:34.481834 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:34.482664 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 00:44:34.482942 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ea834c8f8141aeaeabfd81578a84bb; no voters:
I20260927 00:44:34.483178 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:34.483316 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:34.483541 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [term 1 LEADER]: Becoming Leader. State: Replica: 50ea834c8f8141aeaeabfd81578a84bb, State: Running, Role: LEADER
I20260927 00:44:34.483876 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 00:44:34.484310 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:34.485566 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ea834c8f8141aeaeabfd81578a84bb. Latest consensus state: current_term: 1 leader_uuid: "50ea834c8f8141aeaeabfd81578a84bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260927 00:44:34.485612 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ea834c8f8141aeaeabfd81578a84bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ea834c8f8141aeaeabfd81578a84bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260927 00:44:34.485795 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:34.485694 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:34.486670 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:34.489745 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:34.495209 3509 catalog_manager.cc:1347] Generated new cluster ID: 8eb4552202684dd08223e61474810902
I20260927 00:44:34.495282 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:34.526984 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:34.527936 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:34.532980 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ea834c8f8141aeaeabfd81578a84bb: Generated new TSK 0
I20260927 00:44:34.533663 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:34.562825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:34.671351 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:34.671628 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:34.671705 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:34.675705 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:34.675782 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:34.675869 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:34.680900 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:34.681962 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:34.682958 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:34.693199 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:34.693332 3516 server_base.cc:1034] running on GCE node
W20260927 00:44:34.693297 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:34.693199 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:34.694063 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:34.694725 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:34.695914 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790469874695900 us; error 26 us; skew 500 ppm
Sep 27 00:44:34 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:34.703897 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:34.704811 3516 webserver.cc:460] Webserver started at http://127.0.98.1:35831/ using document root <none> and password file <none>
I20260927 00:44:34.705533 3516 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:34.705633 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:34.705878 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:34.707921 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddf5351975384dd1804e6bd9d2475c94"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "fbced7b0dacac91c39a632e94d3065d7"
server_key_iv: "fe63e3776e72622c28b5d702f37a8cb8"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.708441 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddf5351975384dd1804e6bd9d2475c94"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "fbced7b0dacac91c39a632e94d3065d7"
server_key_iv: "fe63e3776e72622c28b5d702f37a8cb8"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.712177 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:44:34.714536 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:34.715752 3516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:34.715847 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "ddf5351975384dd1804e6bd9d2475c94"
format_stamp: "Formatted at 2026-09-27 00:44:34 on dist-test-slave-l5sb"
server_key: "fbced7b0dacac91c39a632e94d3065d7"
server_key_iv: "fe63e3776e72622c28b5d702f37a8cb8"
server_key_version: "encryptionkey@0"
I20260927 00:44:34.715920 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:34.735555 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:34.738940 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:34.739156 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:34.739758 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:34.740576 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:34.740661 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:34.740736 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:34.740787 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:34.749984 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36823
I20260927 00:44:34.750012 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36823 every 8 connection(s)
I20260927 00:44:34.750911 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:34.759713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3516
I20260927 00:44:34.759847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:34.760224 392 external_mini_cluster.cc:1428] Setting key d1e4fd9af0e0e336138c18c3671a4ffd
Sep 27 00:44:34 dist-test-slave-l5sb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:34.766016 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:34.752718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46685 (local address 127.0.98.62:43803)
0927 00:44:34.752853 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:34.752857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:34.753792 (+ 935us) server_negotiation.cc:407] Connection header received
0927 00:44:34.755211 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:34.755214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:34.755260 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:34.755335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:34.756691 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.757223 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.758483 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.758705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.758781 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:34.761656 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:34.761660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:34.761662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:34.761687 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:34.763535 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:34.764200 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:34.764204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:34.764205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:34.764295 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:34.764740 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:34.764744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:34.764747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:34.764975 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:34.765056 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:34.765744 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 00:44:34.765900 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:34.767198 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:34.753134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.1:46685)
0927 00:44:34.753594 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:34.753655 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:34.755015 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:34.755532 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:34.755543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:34.755987 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:34.756546 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:34.756566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.757325 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.757328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:34.758384 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:34.758394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:34.758863 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:34.758866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:34.759007 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:34.759646 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:34.759668 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:34.761482 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:34.763698 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:34.763706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:34.763722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:34.764086 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:34.764403 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:34.764407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:34.764412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:34.764604 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:34.765143 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:34.765152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:34.765472 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 00:44:34.765695 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 00:44:34.766081 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260927 00:44:34.768519 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260927 00:44:34.768761 3645 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:34.769237 3645 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:34.770723 3454 ts_manager.cc:194] Registered new tserver with Master: ddf5351975384dd1804e6bd9d2475c94 (127.0.98.1:36823)
I20260927 00:44:34.771875 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 00:44:35.112272 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.350s user 0.001s sys 0.002s
I20260927 00:44:35.112843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:35.224745 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:35.225008 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:35.225050 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:35.229079 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:35.229135 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:35.229230 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:35.234724 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:35.235832 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:35.236868 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:35.246402 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:35.246429 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:35.246766 3650 server_base.cc:1034] running on GCE node
W20260927 00:44:35.246407 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:35.247223 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:35.247849 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:35.249035 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790469875249007 us; error 41 us; skew 500 ppm
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:35.251868 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:35.252758 3650 webserver.cc:460] Webserver started at http://127.0.98.2:38431/ using document root <none> and password file <none>
I20260927 00:44:35.253326 3650 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:35.253422 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:35.253659 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:35.255741 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "78689021f5cb4e56838b7d84ef6bf04a"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "78d9c81caa1569efe4ac3cddaf399f3e"
server_key_iv: "d28a3c1069ccdeea5273b77fa422396d"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.256332 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78689021f5cb4e56838b7d84ef6bf04a"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "78d9c81caa1569efe4ac3cddaf399f3e"
server_key_iv: "d28a3c1069ccdeea5273b77fa422396d"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.260651 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:44:35.263248 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.264492 3650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:35.264645 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "78689021f5cb4e56838b7d84ef6bf04a"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "78d9c81caa1569efe4ac3cddaf399f3e"
server_key_iv: "d28a3c1069ccdeea5273b77fa422396d"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.264762 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:35.276078 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:35.278790 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:35.279030 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:35.279623 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:35.280455 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:35.280526 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.280601 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:35.280645 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.289849 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42099
I20260927 00:44:35.289865 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42099 every 8 connection(s)
I20260927 00:44:35.290855 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:35.298949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3650
I20260927 00:44:35.299098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:35.299420 392 external_mini_cluster.cc:1428] Setting key 52f3e236803f43c5ce8616f78513b514
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:35.305078 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:35.292622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44025 (local address 127.0.98.62:43803)
0927 00:44:35.292864 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:35.292868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:35.293565 (+ 697us) server_negotiation.cc:407] Connection header received
0927 00:44:35.295007 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:35.295010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:35.295102 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:35.295205 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:35.296343 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.296886 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.298039 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.298283 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.298396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:35.301166 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:35.301169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:35.301171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:35.301192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:35.302754 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.303363 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.303366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.303367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.303475 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.303833 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.303839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.303843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.304095 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:35.304199 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:35.304672 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 00:44:35.304816 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260927 00:44:35.305840 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:35.292987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.2:44025)
0927 00:44:35.293419 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:35.293460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:35.294787 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:35.295378 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:35.295385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:35.295733 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:35.296188 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:35.296203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.297010 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.297014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:35.297911 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:35.297920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.298430 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.298434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:35.298605 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:35.299303 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:35.299325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:35.300986 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:35.302869 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:35.302876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:35.302892 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:35.303265 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:35.303604 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:35.303606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:35.303607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:35.303740 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:35.304198 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:35.304202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:35.304422 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 00:44:35.304606 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:44:35.304908 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260927 00:44:35.307001 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260927 00:44:35.307248 3779 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:35.307739 3779 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
I20260927 00:44:35.308796 3454 ts_manager.cc:194] Registered new tserver with Master: 78689021f5cb4e56838b7d84ef6bf04a (127.0.98.2:42099)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:35.309352 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:35.326566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:35.436547 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:35.436883 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:35.436962 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:35.441009 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:35.441109 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:35.441205 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:35.446492 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:35.447566 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:35.448611 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:35.458194 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:35.458230 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:35.458204 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:35.458379 3785 server_base.cc:1034] running on GCE node
I20260927 00:44:35.458853 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:35.459474 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:35.460633 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790469875460616 us; error 27 us; skew 500 ppm
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:35.463775 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:35.464648 3785 webserver.cc:460] Webserver started at http://127.0.98.3:44115/ using document root <none> and password file <none>
I20260927 00:44:35.465191 3785 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:35.465279 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:35.465503 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:35.467631 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "50269396b01745edb5e8fe7f4537031e"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "8d94f29a75b76163a2bc3314bdcdf1e1"
server_key_iv: "a4c935750bc86f3feea90fe4deb7efd1"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.468189 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "50269396b01745edb5e8fe7f4537031e"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "8d94f29a75b76163a2bc3314bdcdf1e1"
server_key_iv: "a4c935750bc86f3feea90fe4deb7efd1"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.471974 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:35.474259 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.475397 3785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:35.475524 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "50269396b01745edb5e8fe7f4537031e"
format_stamp: "Formatted at 2026-09-27 00:44:35 on dist-test-slave-l5sb"
server_key: "8d94f29a75b76163a2bc3314bdcdf1e1"
server_key_iv: "a4c935750bc86f3feea90fe4deb7efd1"
server_key_version: "encryptionkey@0"
I20260927 00:44:35.475644 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:35.491707 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:35.494519 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:35.494721 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:35.495285 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:35.496070 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:35.496141 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.496209 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:35.496256 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:35.504837 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38935
I20260927 00:44:35.504859 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38935 every 8 connection(s)
I20260927 00:44:35.505707 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:35.512426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3785
I20260927 00:44:35.512552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:35.512823 392 external_mini_cluster.cc:1428] Setting key a7bed8b05f9d4b498896193e97e7dbcb
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:35.519291 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:35.507551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33287 (local address 127.0.98.62:43803)
0927 00:44:35.507716 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:35.507720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:35.508601 (+ 881us) server_negotiation.cc:407] Connection header received
0927 00:44:35.509960 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:35.509962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:35.510007 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:35.510080 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:35.511159 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.511685 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.513143 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.513433 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.513548 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:35.516016 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:35.516019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:35.516021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:35.516044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:35.517483 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.517973 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.517975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.517977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.518047 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.518335 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.518337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.518338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.518498 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:35.518568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:35.519006 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 00:44:35.519100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:44:35.520035 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:35.508001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.3:33287)
0927 00:44:35.508444 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:35.508486 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:35.509716 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:35.510230 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:35.510237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:35.510578 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:35.511041 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:35.511053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.511788 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.511791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:35.513009 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:35.513019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.513581 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.513585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:35.513824 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:35.514390 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:35.514412 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:35.515875 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:35.517577 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:35.517581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:35.517598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:35.517893 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:35.518127 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:35.518129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:35.518130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:35.518252 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:35.518578 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:35.518582 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:35.518806 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 00:44:35.518948 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 00:44:35.519173 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260927 00:44:35.521101 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260927 00:44:35.521306 3914 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:35.521720 3914 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
I20260927 00:44:35.522675 3454 ts_manager.cc:194] Registered new tserver with Master: 50269396b01745edb5e8fe7f4537031e (127.0.98.3:38935)
I20260927 00:44:35.523169 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33287
I20260927 00:44:35.526649 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:35.551607 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:35.543419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.0.98.62:43803)
0927 00:44:35.543551 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:35.543556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:35.543715 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:44:35.543828 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:35.543831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:35.543874 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:35.543950 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:35.544605 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.545119 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.545983 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:35.546192 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:35.546268 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:35.548439 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:35.548442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:35.548444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:35.548466 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:35.549967 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.550310 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.550313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.550315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.550385 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:35.550595 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:35.550597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:35.550599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:35.550905 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:35.551053 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:35.551285 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:44:35.551401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:44:35.552037 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52850
I20260927 00:44:35.552824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3516
I20260927 00:44:35.558566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3650
I20260927 00:44:35.563635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3785
I20260927 00:44:35.568847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3422
2026-09-27T00:44:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2504 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3926](info): set up 2 sockets
Sep 27 00:44:35 dist-test-slave-l5sb krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260927 00:44:36.609886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:36 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:36.728977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:36.838380 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:36.838630 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:36.838670 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:36.842547 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:36.842593 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:36.842612 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:36.842628 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:36.842643 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:36.847653 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:36.848591 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:36.849519 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:36.858022 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:36.858084 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:36.858093 3939 server_base.cc:1034] running on GCE node
W20260927 00:44:36.858042 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:36.858888 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:36.859952 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:36.861172 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790469876861159 us; error 26 us; skew 500 ppm
Sep 27 00:44:36 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:36.864711 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:36.865891 3939 webserver.cc:460] Webserver started at http://127.0.98.62:38657/ using document root <none> and password file <none>
I20260927 00:44:36.866636 3939 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:36.866734 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:36.866990 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:36.869879 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "7b687e55a2e94840990e00387970495d"
format_stamp: "Formatted at 2026-09-27 00:44:36 on dist-test-slave-l5sb"
server_key: "3956cf9d73561145e8b0a82dfdc00d59"
server_key_iv: "f97a096c719965127e60a8e8410bcff1"
server_key_version: "encryptionkey@0"
I20260927 00:44:36.870649 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b687e55a2e94840990e00387970495d"
format_stamp: "Formatted at 2026-09-27 00:44:36 on dist-test-slave-l5sb"
server_key: "3956cf9d73561145e8b0a82dfdc00d59"
server_key_iv: "f97a096c719965127e60a8e8410bcff1"
server_key_version: "encryptionkey@0"
I20260927 00:44:36.875766 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:36.878993 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:36.881026 3939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:36.881193 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "7b687e55a2e94840990e00387970495d"
format_stamp: "Formatted at 2026-09-27 00:44:36 on dist-test-slave-l5sb"
server_key: "3956cf9d73561145e8b0a82dfdc00d59"
server_key_iv: "f97a096c719965127e60a8e8410bcff1"
server_key_version: "encryptionkey@0"
I20260927 00:44:36.881361 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:36.901432 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:36.904919 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:36.905112 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:36.911927 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42827
I20260927 00:44:36.911949 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42827 every 8 connection(s)
I20260927 00:44:36.912832 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:36.914958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 3939
I20260927 00:44:36.915091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:36.915359 392 external_mini_cluster.cc:1428] Setting key 137ce5b7597c3b6fc29a8207d7ea2773
I20260927 00:44:36.915751 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:36.921631 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: Bootstrap starting.
Sep 27 00:44:36 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:36.923991 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:36.924772 4007 log.cc:824] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:36.926900 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: No bootstrap required, opened a new log
I20260927 00:44:36.928920 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:36.916689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.98.62:42827)
0927 00:44:36.917126 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:36.917140 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:36.917174 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:44:36.917832 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:36.917853 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:36.918174 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:36.918492 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:36.919282 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:36.920253 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:36.921091 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:36.921556 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:36.921734 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:36.924381 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:36.924386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:36.924399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:36.924424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:36.926222 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:36.926705 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:36.926709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:36.926713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:36.926813 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:36.927106 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:36.927109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:36.927110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:36.927480 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:36.927625 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:36.927910 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:44:36.928195 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260927 00:44:36.930866 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260927 00:44:36.931177 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:36.931249 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b687e55a2e94840990e00387970495d, State: Initialized, Role: FOLLOWER
I20260927 00:44:36.931648 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260927 00:44:36.931788 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:36.931855 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:36.931979 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:36.932860 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260927 00:44:36.933212 4007 leader_election.cc:302] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b687e55a2e94840990e00387970495d; no voters:
I20260927 00:44:36.933537 4007 leader_election.cc:288] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:36.933692 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:36.933998 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [term 1 LEADER]: Becoming Leader. State: Replica: 7b687e55a2e94840990e00387970495d, State: Running, Role: LEADER
I20260927 00:44:36.934355 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260927 00:44:36.935111 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:36.936138 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b687e55a2e94840990e00387970495d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } } }
I20260927 00:44:36.936131 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b687e55a2e94840990e00387970495d. Latest consensus state: current_term: 1 leader_uuid: "7b687e55a2e94840990e00387970495d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b687e55a2e94840990e00387970495d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } } }
I20260927 00:44:36.936278 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:36.936278 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d [sys.catalog]: This master's current role is: LEADER
W20260927 00:44:36.941067 4028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:36.941207 4028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:36.941310 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:36.942236 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:36.951045 4021 catalog_manager.cc:1347] Generated new cluster ID: 282c4c3effa8461c8ae1cb6222c5a399
I20260927 00:44:36.951122 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:36.961205 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:36.962286 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:36.969004 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b687e55a2e94840990e00387970495d: Generated new TSK 0
I20260927 00:44:36.969894 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:37.002866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:37.110574 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:37.110868 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:37.110940 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:37.114910 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:37.115000 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:37.115100 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:37.120183 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:37.121208 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:37.122231 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:37.131307 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:37.131325 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:37.131433 4033 server_base.cc:1034] running on GCE node
W20260927 00:44:37.131307 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:37.132022 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:37.132653 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:37.133845 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790469877133827 us; error 29 us; skew 500 ppm
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:37.136598 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:37.137543 4033 webserver.cc:460] Webserver started at http://127.0.98.1:35447/ using document root <none> and password file <none>
I20260927 00:44:37.138154 4033 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:37.138216 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:37.138447 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:37.140427 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "371048c8503b4db7be6a84558b8d2f41"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "c28687c3d637b6b8f84cb32a1f6bfc54"
server_key_iv: "e5fa9778bd94de9e220f66c3a16976b2"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.140985 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "371048c8503b4db7be6a84558b8d2f41"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "c28687c3d637b6b8f84cb32a1f6bfc54"
server_key_iv: "e5fa9778bd94de9e220f66c3a16976b2"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.144510 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:37.146865 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.148131 4033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:37.148263 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "371048c8503b4db7be6a84558b8d2f41"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "c28687c3d637b6b8f84cb32a1f6bfc54"
server_key_iv: "e5fa9778bd94de9e220f66c3a16976b2"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.148370 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:37.177172 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:37.180606 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:37.180814 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:37.181422 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:37.182298 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:37.182376 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.182456 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:37.182507 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.192126 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44485
I20260927 00:44:37.192149 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44485 every 8 connection(s)
I20260927 00:44:37.193022 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:37.199219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4033
I20260927 00:44:37.199362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:37.199687 392 external_mini_cluster.cc:1428] Setting key e8acade9fc1d9c92d26699003541d67e
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:37.208158 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.194881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47669 (local address 127.0.98.62:42827)
0927 00:44:37.195018 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:37.195022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:37.195948 (+ 926us) server_negotiation.cc:407] Connection header received
0927 00:44:37.197228 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:37.197231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:37.197276 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:37.197349 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:37.198428 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.199016 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.200633 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.200980 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.201092 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.204200 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:37.204207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:37.204213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:37.204246 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:37.206009 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.206506 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.206513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.206516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.206615 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.206940 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.206946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.206950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.207230 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:37.207340 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:37.207811 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 00:44:37.207950 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:37.208868 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.195260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.1:47669)
0927 00:44:37.195791 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:37.195832 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:37.197041 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:37.197469 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:37.197475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:37.197835 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:37.198294 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.198306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.199156 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.199160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.200514 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.200523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.201153 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.201157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.201340 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.202053 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:37.202079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:37.204021 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:37.206136 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.206141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.206155 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.206401 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.206716 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.206719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.206720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.206846 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.207339 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:37.207344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:37.207571 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 00:44:37.207721 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:44:37.207962 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:37.210188 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260927 00:44:37.210438 4162 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:37.210992 4162 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:37.212716 3971 ts_manager.cc:194] Registered new tserver with Master: 371048c8503b4db7be6a84558b8d2f41 (127.0.98.1:44485)
I20260927 00:44:37.214260 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:37.229732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:37.340004 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:37.340246 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:37.340319 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:37.344341 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:37.344400 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:37.344463 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:37.349653 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:37.350757 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:37.351760 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:37.361500 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:37.361500 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:37.361562 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:37.361688 4167 server_base.cc:1034] running on GCE node
I20260927 00:44:37.362092 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:37.362697 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:37.363879 4167 hybrid_clock.cc:648] HybridClock initialized: now 1790469877363846 us; error 43 us; skew 500 ppm
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:37.366567 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:37.367388 4167 webserver.cc:460] Webserver started at http://127.0.98.2:44505/ using document root <none> and password file <none>
I20260927 00:44:37.367937 4167 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:37.368023 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:37.368259 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:37.370406 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "223c640b262a4a2d9b6ba4d25990db52"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "44e3de172d3839c9d3acbaf6eed31851"
server_key_iv: "9afc045884988a89c7f97dc715d548b6"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.370934 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "223c640b262a4a2d9b6ba4d25990db52"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "44e3de172d3839c9d3acbaf6eed31851"
server_key_iv: "9afc045884988a89c7f97dc715d548b6"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.374449 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:44:37.376782 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.378008 4167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 00:44:37.378151 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "223c640b262a4a2d9b6ba4d25990db52"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "44e3de172d3839c9d3acbaf6eed31851"
server_key_iv: "9afc045884988a89c7f97dc715d548b6"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.378257 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:37.406831 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:37.410254 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:37.410459 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:37.411048 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:37.411865 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:37.411933 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.412006 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:37.412056 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.420697 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41731
I20260927 00:44:37.420722 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41731 every 8 connection(s)
I20260927 00:44:37.421581 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:37.426070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4167
I20260927 00:44:37.426205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:37.426497 392 external_mini_cluster.cc:1428] Setting key 6ec9f43d071213e3f98690dcc4f9327b
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:37.437047 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.423323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54689 (local address 127.0.98.62:42827)
0927 00:44:37.423455 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:37.423459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:37.424206 (+ 747us) server_negotiation.cc:407] Connection header received
0927 00:44:37.425609 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:37.425612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:37.425671 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:37.425778 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:37.427141 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.427905 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.429660 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.430081 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.430196 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.432901 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:37.432908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:37.432913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:37.432947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:37.434673 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.435237 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.435242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.435244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.435332 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.435760 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.435764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.435766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.436045 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:37.436169 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:37.436754 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 00:44:37.436882 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:44:37.438192 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.423640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.2:54689)
0927 00:44:37.424057 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:37.424100 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:37.425377 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:37.425978 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:37.425985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:37.426448 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:37.426969 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.427005 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.428035 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.428038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.429495 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.429509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.430220 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.430223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.430365 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.430998 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:37.431014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:37.432724 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:37.434780 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.434784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.434797 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.435140 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.435426 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.435428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.435430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.435632 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.436176 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:37.436184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:37.436426 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 00:44:37.436691 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 00:44:37.437065 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:44:37.439432 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260927 00:44:37.439689 4296 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:37.440172 4296 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
I20260927 00:44:37.441226 3971 ts_manager.cc:194] Registered new tserver with Master: 223c640b262a4a2d9b6ba4d25990db52 (127.0.98.2:41731)
I20260927 00:44:37.441854 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:37.452817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:37.560781 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:37.560995 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:37.561034 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:37.564845 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:37.564888 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:37.564941 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:37.569875 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:37.570762 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:37.571650 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:37.580878 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:37.580839 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:37.580878 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:37.580929 4301 server_base.cc:1034] running on GCE node
I20260927 00:44:37.581492 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:37.582252 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:37.583434 4301 hybrid_clock.cc:648] HybridClock initialized: now 1790469877583390 us; error 64 us; skew 500 ppm
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:37.586095 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:37.586988 4301 webserver.cc:460] Webserver started at http://127.0.98.3:40401/ using document root <none> and password file <none>
I20260927 00:44:37.587543 4301 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:37.587648 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:37.587910 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:37.589983 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed0480a0343e47a299d9692944660759"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "0c6d1bea3c6a8a0506d00204602d11eb"
server_key_iv: "7af24d7f15c3bc54b6b12863b5bf3a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.590536 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed0480a0343e47a299d9692944660759"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "0c6d1bea3c6a8a0506d00204602d11eb"
server_key_iv: "7af24d7f15c3bc54b6b12863b5bf3a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.594076 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:44:37.596309 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.597604 4301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:37.597808 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "ed0480a0343e47a299d9692944660759"
format_stamp: "Formatted at 2026-09-27 00:44:37 on dist-test-slave-l5sb"
server_key: "0c6d1bea3c6a8a0506d00204602d11eb"
server_key_iv: "7af24d7f15c3bc54b6b12863b5bf3a55"
server_key_version: "encryptionkey@0"
I20260927 00:44:37.597944 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:37.611594 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:37.614445 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:37.614650 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:37.615231 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:37.616071 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:37.616147 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.616216 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:37.616255 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:37.625533 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32869
I20260927 00:44:37.625555 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32869 every 8 connection(s)
I20260927 00:44:37.626483 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:37.628675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4301
I20260927 00:44:37.628834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:37.629168 392 external_mini_cluster.cc:1428] Setting key 264731c01640a02f2cfa282e4a073bc1
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:37.640331 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.628191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59257 (local address 127.0.98.62:42827)
0927 00:44:37.628335 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:37.628344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:37.629220 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:44:37.631118 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:37.631121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:37.631181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:37.631285 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:37.632479 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.632982 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.634229 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.634431 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.634504 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.637064 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:37.637067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:37.637069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:37.637093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:37.638454 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.638891 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.638894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.638896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.638976 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.639317 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.639319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.639320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.639550 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:37.639633 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:37.640098 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:44:37.640207 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:44:37.641171 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.628500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.3:59257)
0927 00:44:37.629024 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:37.629085 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:37.630921 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:37.631507 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:37.631517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:37.631880 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:37.632357 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.632369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.633094 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.633096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.634135 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:37.634145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.634561 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.634563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:37.634718 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.635401 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:37.635423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:37.636921 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:37.638555 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.638559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.638569 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.638805 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.639086 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:37.639089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:37.639090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:37.639223 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:37.639657 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:37.639662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:37.639876 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:44:37.640026 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:44:37.640260 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260927 00:44:37.642401 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260927 00:44:37.642658 4430 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:37.643179 4430 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
I20260927 00:44:37.644079 3971 ts_manager.cc:194] Registered new tserver with Master: ed0480a0343e47a299d9692944660759 (127.0.98.3:32869)
I20260927 00:44:37.644596 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59257
I20260927 00:44:37.654686 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:37.678799 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.670124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.0.98.1:44485)
0927 00:44:37.670428 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:37.670444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:37.670474 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:44:37.670523 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:37.670528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:37.670725 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:37.670842 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:37.671456 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.672219 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.673039 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.673399 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.673479 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.675782 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:37.675790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:37.675803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:37.675825 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:37.677199 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.677562 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.677567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.677569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.677652 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.677917 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.677919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.677920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.678137 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:37.678292 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:37.678440 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:44:37.678603 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":135,"threads_started":1}
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:37.689153 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:37.681533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.0.98.62:42827)
0927 00:44:37.681666 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:37.681673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:37.681754 (+ 81us) server_negotiation.cc:407] Connection header received
0927 00:44:37.681942 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:37.681945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:37.681997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:37.682099 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:37.682692 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.683204 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.683961 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:37.684192 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:37.684299 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:37.686489 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:37.686492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:37.686494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:37.686518 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:37.687775 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.688150 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.688152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.688153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.688223 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:37.688441 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:37.688444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:37.688445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:37.688657 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:37.688796 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:37.688948 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:44:37.689038 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:44:37.689502 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49928
I20260927 00:44:37.690127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4033
I20260927 00:44:37.696189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4167
I20260927 00:44:37.701342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4301
I20260927 00:44:37.706503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 3939
2026-09-27T00:44:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2137 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[4442](info): set up 2 sockets
Sep 27 00:44:37 dist-test-slave-l5sb krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260927 00:44:38.737986 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:38 dist-test-slave-l5sb krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:38.870568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:38.983946 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:38.984385 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:38.984505 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:38.991330 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:38.991432 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:38.991488 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:38.991534 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:38.991571 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:39.000171 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:39.001549 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.002593 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.010704 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.010696 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.010696 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.011142 4455 server_base.cc:1034] running on GCE node
I20260927 00:44:39.011687 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.012820 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.013998 4455 hybrid_clock.cc:648] HybridClock initialized: now 1790469879013975 us; error 48 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.016733 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:39.017508 4455 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.017601 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.017879 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:39.019989 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "24a96bc489f641bb85b4d29e9a997e18"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "92e8155c670ab082c3fd28cc770ad4ec"
server_key_iv: "40276e018fc6c87ada8f66490ff878ce"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.020565 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "24a96bc489f641bb85b4d29e9a997e18"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "92e8155c670ab082c3fd28cc770ad4ec"
server_key_iv: "40276e018fc6c87ada8f66490ff878ce"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.024165 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:39.026578 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.027755 4455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:39.027889 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "24a96bc489f641bb85b4d29e9a997e18"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "92e8155c670ab082c3fd28cc770ad4ec"
server_key_iv: "40276e018fc6c87ada8f66490ff878ce"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.027998 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.050767 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.053694 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.053936 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.060889 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33409
I20260927 00:44:39.061172 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33409 every 8 connection(s)
I20260927 00:44:39.061538 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:39.064200 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.066599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4455
I20260927 00:44:39.066759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:39.067039 392 external_mini_cluster.cc:1428] Setting key b8c23f764d209aa8e9d702e65d20fec6
I20260927 00:44:39.070477 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: Bootstrap starting.
I20260927 00:44:39.073101 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.073930 4522 log.cc:824] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.076318 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: No bootstrap required, opened a new log
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.080813 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260927 00:44:39.081067 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.081102 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a96bc489f641bb85b4d29e9a997e18, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.081643 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260927 00:44:39.081831 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:39.081904 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:39.082005 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.083237 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260927 00:44:39.083609 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a96bc489f641bb85b4d29e9a997e18; no voters:
I20260927 00:44:39.083950 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:39.083974 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.068511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.98.62:33409)
0927 00:44:39.068923 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.068941 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.068985 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:44:39.069758 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.069812 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.070234 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.070592 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.071318 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.072755 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.073728 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.074452 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.074574 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.077979 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.077985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.078005 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.078040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.080690 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.081227 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.081232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.081237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.081392 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.081738 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.081741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.081742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.082350 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.082521 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.082802 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.083050 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260927 00:44:39.084025 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:39.084210 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [term 1 LEADER]: Becoming Leader. State: Replica: 24a96bc489f641bb85b4d29e9a997e18, State: Running, Role: LEADER
I20260927 00:44:39.084506 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260927 00:44:39.085271 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:39.085734 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a96bc489f641bb85b4d29e9a997e18. Latest consensus state: current_term: 1 leader_uuid: "24a96bc489f641bb85b4d29e9a997e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260927 00:44:39.086499 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:39.086288 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a96bc489f641bb85b4d29e9a997e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a96bc489f641bb85b4d29e9a997e18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260927 00:44:39.086709 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:39.086943 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:39.089917 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:39.095060 4536 catalog_manager.cc:1347] Generated new cluster ID: 77770d56f8254c28b93ffb10fde01e04
I20260927 00:44:39.095122 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:39.125969 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:39.126987 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:39.137537 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: Generated new TSK 0
I20260927 00:44:39.138267 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:39.181105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:39.293344 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.293864 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.293929 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.298671 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.298743 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.298810 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:39.304232 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:39.305267 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.306398 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.315781 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.315807 4548 server_base.cc:1034] running on GCE node
W20260927 00:44:39.315815 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.316015 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.316439 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.317080 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.318264 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790469879318242 us; error 32 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.321113 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:39.321918 4548 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.322013 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.322254 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:39.324312 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "9193bdd485ae4528ae0d027393424a5b"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "d8cfb8187d680fc66776e7cf7be5ae4e"
server_key_iv: "da26c2c3ee56cf0b3219507ee5ed50e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.324887 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9193bdd485ae4528ae0d027393424a5b"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "d8cfb8187d680fc66776e7cf7be5ae4e"
server_key_iv: "da26c2c3ee56cf0b3219507ee5ed50e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.328691 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:44:39.331239 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.332556 4548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:39.332701 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "9193bdd485ae4528ae0d027393424a5b"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "d8cfb8187d680fc66776e7cf7be5ae4e"
server_key_iv: "da26c2c3ee56cf0b3219507ee5ed50e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.332813 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.359606 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.362907 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.363117 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.363706 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.364523 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:39.364596 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.364656 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:39.364705 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.374145 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44085
I20260927 00:44:39.374179 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44085 every 8 connection(s)
I20260927 00:44:39.374764 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:39.377671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4548
I20260927 00:44:39.377817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:39.378113 392 external_mini_cluster.cc:1428] Setting key f2e59232574225ec4d5ccde551cf8464
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:39.390511 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.376653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52137 (local address 127.0.98.62:33409)
0927 00:44:39.376828 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.376833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.377673 (+ 840us) server_negotiation.cc:407] Connection header received
0927 00:44:39.379551 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.379554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.379601 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.379694 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.381280 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.382210 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.383412 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.383643 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.383753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.386794 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.386801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.386808 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.386837 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.388427 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.388940 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.388945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.388946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.389029 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.389401 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.389405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.389407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.389584 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.389685 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.390230 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.390349 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:44:39.391508 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.377058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.1:52137)
0927 00:44:39.377513 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.377556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.379296 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.379883 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.379892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.380438 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.381130 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.381152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.382336 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.382339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.383265 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.383279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.383788 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.383793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.383964 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.384571 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.384597 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.386591 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.388543 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.388549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.388565 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.388828 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.389129 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.389138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.389140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.389295 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.389698 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.389706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.390006 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.390193 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.390529 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260927 00:44:39.392824 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260927 00:44:39.393085 4676 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.393535 4676 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260927 00:44:39.395150 4486 ts_manager.cc:194] Registered new tserver with Master: 9193bdd485ae4528ae0d027393424a5b (127.0.98.1:44085)
I20260927 00:44:39.396283 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:39.406962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:39.514430 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.514700 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.514772 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.518699 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.518783 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.518870 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:39.523957 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:39.524932 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.525905 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.535519 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.535532 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.535620 4681 server_base.cc:1034] running on GCE node
W20260927 00:44:39.535530 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.536180 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.536821 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.538018 4681 hybrid_clock.cc:648] HybridClock initialized: now 1790469879537998 us; error 34 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.541249 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:39.542127 4681 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.542229 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.542459 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:39.544673 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "06fa8cca78954d238714f18f05f83a19"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "c92a4523c0797b09ad5328de5ae387a2"
server_key_iv: "3f223d3c69b00cba189f456fdbd5414c"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.545307 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "06fa8cca78954d238714f18f05f83a19"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "c92a4523c0797b09ad5328de5ae387a2"
server_key_iv: "3f223d3c69b00cba189f456fdbd5414c"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.549230 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:44:39.551910 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.553153 4681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:39.553296 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "06fa8cca78954d238714f18f05f83a19"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "c92a4523c0797b09ad5328de5ae387a2"
server_key_iv: "3f223d3c69b00cba189f456fdbd5414c"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.553462 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.573102 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.576228 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.576437 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.577039 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.577921 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:39.578003 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.578065 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:39.578116 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.587173 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260927 00:44:39.587206 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260927 00:44:39.587770 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:39.593159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4681
I20260927 00:44:39.593287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:39.593564 392 external_mini_cluster.cc:1428] Setting key e3006f09ea535123877902f470c9ad88
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.602067 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.589396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59371 (local address 127.0.98.62:33409)
0927 00:44:39.589524 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.589528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.590372 (+ 844us) server_negotiation.cc:407] Connection header received
0927 00:44:39.591780 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.591783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.591835 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.591913 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.593048 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.593717 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.595274 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.595508 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.595615 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.598071 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.598078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.598084 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.598125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.599941 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.600435 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.600442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.600445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.600552 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.600869 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.600876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.600879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.601139 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.601257 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.601706 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.601871 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:39.602837 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.589801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.2:59371)
0927 00:44:39.590233 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.590273 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.591575 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.592081 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.592087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.592438 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.592882 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.592903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.593887 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.593891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.595143 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.595152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.595650 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.595654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.595823 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.596415 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.596431 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.597911 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.600071 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.600076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.600087 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.600325 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.600651 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.600653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.600655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.600776 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.601247 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.601251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.601467 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.601613 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.601917 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:39.604036 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260927 00:44:39.604286 4809 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.604732 4809 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260927 00:44:39.605762 4486 ts_manager.cc:194] Registered new tserver with Master: 06fa8cca78954d238714f18f05f83a19 (127.0.98.2:37875)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:44:39.606601 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:39.621471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:39.736042 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.736315 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.736390 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.740530 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.740584 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.740643 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:39.745687 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:39.746682 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.747611 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.756524 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.756546 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.756717 4814 server_base.cc:1034] running on GCE node
W20260927 00:44:39.756891 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.757290 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.758006 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.759184 4814 hybrid_clock.cc:648] HybridClock initialized: now 1790469879759171 us; error 26 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.767622 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:39.768427 4814 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.768520 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.768748 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:39.770882 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed897cb4e49e4b9e8516c44bb66d4f60"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "48e0e1de5eb78f070ead8d4905a1afd0"
server_key_iv: "ca31d4c3f1ba4af04cb85eab6c005579"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.771477 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed897cb4e49e4b9e8516c44bb66d4f60"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "48e0e1de5eb78f070ead8d4905a1afd0"
server_key_iv: "ca31d4c3f1ba4af04cb85eab6c005579"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.775094 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:39.777596 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.778891 4814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:39.779021 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "ed897cb4e49e4b9e8516c44bb66d4f60"
format_stamp: "Formatted at 2026-09-27 00:44:39 on dist-test-slave-l5sb"
server_key: "48e0e1de5eb78f070ead8d4905a1afd0"
server_key_iv: "ca31d4c3f1ba4af04cb85eab6c005579"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.779127 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.800488 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.803483 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.803686 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.804253 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.805038 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:39.805110 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.805182 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:39.805225 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.814288 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41537
I20260927 00:44:39.814337 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41537 every 8 connection(s)
I20260927 00:44:39.814884 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:39.817658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4814
I20260927 00:44:39.817799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:39.818127 392 external_mini_cluster.cc:1428] Setting key 62cacbf4749da52d2487a7632f8b85fa
Sep 27 00:44:39 dist-test-slave-l5sb krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.828738 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.816634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33599 (local address 127.0.98.62:33409)
0927 00:44:39.816781 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.816785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.817690 (+ 905us) server_negotiation.cc:407] Connection header received
0927 00:44:39.819554 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.819556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.819628 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.819718 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.821007 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.821537 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.822715 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.822935 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.823019 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.825457 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.825461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.825463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.825486 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.826818 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.827275 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.827277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.827279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.827384 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.827719 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.827721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.827722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.827900 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.827978 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.828499 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.828614 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:44:39.829522 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.817001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.3:33599)
0927 00:44:39.817494 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.817555 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.819282 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.819878 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.819885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.820380 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.820878 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.820892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.821637 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.821640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.822592 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.822601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.823061 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.823064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.823198 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.823750 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.823766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.825289 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.826926 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.826931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.826943 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.827185 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.827511 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.827513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.827514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.827638 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.827988 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.827992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.828213 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.828367 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.828651 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260927 00:44:39.830910 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260927 00:44:39.831171 4942 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.831851 4942 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260927 00:44:39.832913 4486 ts_manager.cc:194] Registered new tserver with Master: ed897cb4e49e4b9e8516c44bb66d4f60 (127.0.98.3:41537)
I20260927 00:44:39.833551 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33599
I20260927 00:44:39.844214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:39.853557 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.846138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.0.98.62:33409)
0927 00:44:39.846284 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.846288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.846499 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:44:39.846567 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.846569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.846609 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.846674 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.847414 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.847941 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.848843 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.849085 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.849170 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.850621 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.850624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.850626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.850651 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.851953 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.852354 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.852357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.852358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.852434 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.852655 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.852657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.852658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.852859 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.852960 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.853213 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.853368 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:44:39.856832 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:39.860208 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:39.874559 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.869000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:52412)
0927 00:44:39.870087 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.870101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.870188 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.870698 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.870701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.870715 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.870815 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.870821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.871843 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.871849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.873105 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.873114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.874031 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.874139 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.874177 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.874289 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.874406 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":90,"threads_started":1}
I20260927 00:44:39.874617 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.869499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.0.98.1:44085)
0927 00:44:39.869917 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.869924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.870198 (+ 274us) server_negotiation.cc:407] Connection header received
0927 00:44:39.870284 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.870289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.870476 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.870643 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.870938 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.871711 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.873235 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.873912 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874021 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.874161 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.874348 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.874453 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":90,"threads_started":1}
I20260927 00:44:39.875372 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.868664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:55232)
0927 00:44:39.869757 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.869793 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.869886 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.870532 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.870538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.870560 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.870702 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.870708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.872176 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.872179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.873591 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.873601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.874896 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.874988 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.875045 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.875131 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.875205 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":141,"thread_start_us":91,"threads_started":1}
I20260927 00:44:39.875427 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.868631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41537 (local address 127.0.0.1:45100)
0927 00:44:39.868971 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.869020 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.869139 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.870805 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.870808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.870834 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.870945 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.870952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.872302 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.872305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.873591 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.873601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.874937 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.875069 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.875087 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.875179 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.875253 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":96,"thread_start_us":109,"threads_started":1}
I20260927 00:44:39.875836 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.869320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.0.98.3:41537)
0927 00:44:39.870191 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.870202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.870222 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:44:39.870285 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.870289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.870476 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.870643 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.871080 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.872169 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.873731 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.874565 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874671 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.874876 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.875587 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.875688 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":150,"threads_started":1}
I20260927 00:44:39.876045 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.869630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.0.98.2:37875)
0927 00:44:39.870039 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.870054 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.870076 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:44:39.870158 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.870165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.870366 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.870533 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.870829 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.872029 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.873759 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.874586 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.874700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.874840 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.875181 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.875907 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":94,"threads_started":1}
I20260927 00:44:39.878350 4743 tablet_service.cc:1467] Processing CreateTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 (DEFAULT_TABLE table=test-table [id=44c186f37eba4f2c94a84ae468115ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.878458 4876 tablet_service.cc:1467] Processing CreateTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 (DEFAULT_TABLE table=test-table [id=44c186f37eba4f2c94a84ae468115ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.878314 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 (DEFAULT_TABLE table=test-table [id=44c186f37eba4f2c94a84ae468115ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.879311 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cfd4672f4b49afb26892af5b8cdd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.879359 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cfd4672f4b49afb26892af5b8cdd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.879354 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cfd4672f4b49afb26892af5b8cdd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.886277 4959 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Bootstrap starting.
I20260927 00:44:39.887194 4961 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: Bootstrap starting.
I20260927 00:44:39.888497 4959 tablet_bootstrap.cc:654] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.888880 4961 tablet_bootstrap.cc:654] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.889593 4959 log.cc:824] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.889997 4960 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Bootstrap starting.
I20260927 00:44:39.889982 4961 log.cc:824] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.891336 4959 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: No bootstrap required, opened a new log
I20260927 00:44:39.891546 4959 ts_tablet_manager.cc:1397] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:39.891700 4960 tablet_bootstrap.cc:654] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.892797 4960 log.cc:824] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.894457 4960 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: No bootstrap required, opened a new log
I20260927 00:44:39.894632 4960 ts_tablet_manager.cc:1397] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:39.894831 4959 raft_consensus.cc:348] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.895081 4959 raft_consensus.cc:374] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.895159 4959 raft_consensus.cc:729] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fa8cca78954d238714f18f05f83a19, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.895586 4959 consensus_queue.cc:260] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.896257 4809 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260927 00:44:39.896531 4959 ts_tablet_manager.cc:1428] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:44:39.898000 4960 raft_consensus.cc:348] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.898218 4960 raft_consensus.cc:374] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.898262 4961 tablet_bootstrap.cc:492] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: No bootstrap required, opened a new log
I20260927 00:44:39.898268 4960 raft_consensus.cc:729] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed897cb4e49e4b9e8516c44bb66d4f60, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.898461 4961 ts_tablet_manager.cc:1397] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260927 00:44:39.898685 4960 consensus_queue.cc:260] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.899641 4960 ts_tablet_manager.cc:1428] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:44:39.899947 4942 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260927 00:44:39.901894 4961 raft_consensus.cc:348] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.902118 4961 raft_consensus.cc:374] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.902176 4961 raft_consensus.cc:729] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9193bdd485ae4528ae0d027393424a5b, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.902558 4961 consensus_queue.cc:260] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.903103 4676 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260927 00:44:39.903350 4961 ts_tablet_manager.cc:1428] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:44:39.928301 4965 raft_consensus.cc:482] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:39.928512 4965 raft_consensus.cc:504] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.929690 4965 leader_election.cc:288] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed897cb4e49e4b9e8516c44bb66d4f60 (127.0.98.3:41537), 9193bdd485ae4528ae0d027393424a5b (127.0.98.1:44085)
I20260927 00:44:39.934965 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.930042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46411 (local address 127.0.98.3:41537)
0927 00:44:39.930190 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.930195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.930577 (+ 382us) server_negotiation.cc:407] Connection header received
0927 00:44:39.930770 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.930773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.930820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.930901 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.931244 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.931982 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.933562 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.934224 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.934339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.934404 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.934760 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.934836 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:44:39.934965 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.930061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48353 (local address 127.0.98.1:44085)
0927 00:44:39.930189 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.930194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.930338 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:44:39.930486 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.930490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.930554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.930627 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.931047 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.931870 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.933562 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.934233 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.934339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.934404 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.934760 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.934837 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:39.934973 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.929988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.98.2:48353)
0927 00:44:39.930243 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.930259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.930367 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.930776 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.930779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.930797 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.930941 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.930949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.931994 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.931998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.933395 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.933405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.934347 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.934351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.934565 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.934624 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.934704 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.934805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":160}
I20260927 00:44:39.934986 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.930055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41537 (local address 127.0.98.2:46411)
0927 00:44:39.930492 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.930507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.930645 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.930930 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.930932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.930950 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.931127 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.931134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.932069 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.932071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.933395 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.933405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.934347 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.934351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.934561 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.934624 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.934704 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.934805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":65,"thread_start_us":136,"threads_started":1}
I20260927 00:44:39.935662 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cfd4672f4b49afb26892af5b8cdd05" candidate_uuid: "06fa8cca78954d238714f18f05f83a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9193bdd485ae4528ae0d027393424a5b" is_pre_election: true
I20260927 00:44:39.935781 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cfd4672f4b49afb26892af5b8cdd05" candidate_uuid: "06fa8cca78954d238714f18f05f83a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" is_pre_election: true
I20260927 00:44:39.936056 4896 raft_consensus.cc:2454] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06fa8cca78954d238714f18f05f83a19 in term 0.
I20260927 00:44:39.936017 4630 raft_consensus.cc:2454] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06fa8cca78954d238714f18f05f83a19 in term 0.
I20260927 00:44:39.936558 4696 leader_election.cc:302] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06fa8cca78954d238714f18f05f83a19, 9193bdd485ae4528ae0d027393424a5b, ed897cb4e49e4b9e8516c44bb66d4f60; no voters:
I20260927 00:44:39.936834 4965 raft_consensus.cc:2781] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:39.936933 4965 raft_consensus.cc:482] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:39.936995 4965 raft_consensus.cc:3037] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.938128 4965 raft_consensus.cc:504] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.938513 4965 leader_election.cc:288] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [CANDIDATE]: Term 1 election: Requested vote from peers ed897cb4e49e4b9e8516c44bb66d4f60 (127.0.98.3:41537), 9193bdd485ae4528ae0d027393424a5b (127.0.98.1:44085)
I20260927 00:44:39.938858 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cfd4672f4b49afb26892af5b8cdd05" candidate_uuid: "06fa8cca78954d238714f18f05f83a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60"
I20260927 00:44:39.938876 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cfd4672f4b49afb26892af5b8cdd05" candidate_uuid: "06fa8cca78954d238714f18f05f83a19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9193bdd485ae4528ae0d027393424a5b"
I20260927 00:44:39.938984 4630 raft_consensus.cc:3037] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.938985 4896 raft_consensus.cc:3037] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.940265 4896 raft_consensus.cc:2454] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06fa8cca78954d238714f18f05f83a19 in term 1.
I20260927 00:44:39.940287 4630 raft_consensus.cc:2454] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06fa8cca78954d238714f18f05f83a19 in term 1.
I20260927 00:44:39.940595 4696 leader_election.cc:302] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06fa8cca78954d238714f18f05f83a19, ed897cb4e49e4b9e8516c44bb66d4f60; no voters:
I20260927 00:44:39.940791 4965 raft_consensus.cc:2781] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:39.941022 4965 raft_consensus.cc:686] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 1 LEADER]: Becoming Leader. State: Replica: 06fa8cca78954d238714f18f05f83a19, State: Running, Role: LEADER
I20260927 00:44:39.941390 4965 consensus_queue.cc:237] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260927 00:44:39.944425 4486 catalog_manager.cc:5495] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06fa8cca78954d238714f18f05f83a19 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "06fa8cca78954d238714f18f05f83a19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06fa8cca78954d238714f18f05f83a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:39.974352 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.970666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.98.2:37875)
0927 00:44:39.970817 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.970837 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.970865 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:44:39.970954 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.970956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.971011 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.971125 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:39.971426 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.971945 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.972836 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.973058 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.973169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.973497 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:39.973945 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:39.974087 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.974128 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.974184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:44:39.980067 4896 raft_consensus.cc:1261] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 1 FOLLOWER]: Refusing update from remote peer 06fa8cca78954d238714f18f05f83a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:39.980091 4630 raft_consensus.cc:1261] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 1 FOLLOWER]: Refusing update from remote peer 06fa8cca78954d238714f18f05f83a19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:39.980770 4969 consensus_queue.cc:1035] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9193bdd485ae4528ae0d027393424a5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:39.980953 4965 consensus_queue.cc:1035] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed897cb4e49e4b9e8516c44bb66d4f60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:39.988819 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333764628391424000 to 7333764628242833408. Current Snapshot: MvccSnapshot[applied={T|T < 7333764628391424000}]
I20260927 00:44:39.992015 4972 mvcc.cc:205] Tried to move back new op lower bound from 7333764628391424000 to 7333764628242833408. Current Snapshot: MvccSnapshot[applied={T|T < 7333764628391424000 or (T in {7333764628391424000})}]
I20260927 00:44:39.996752 4485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:39.996923 4485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:39.999115 4485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24a96bc489f641bb85b4d29e9a997e18: Sending DeleteTablet for 3 replicas of tablet 12cfd4672f4b49afb26892af5b8cdd05
I20260927 00:44:39.999818 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45100
I20260927 00:44:39.999897 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55232
I20260927 00:44:39.999959 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cfd4672f4b49afb26892af5b8cdd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52412
I20260927 00:44:40.000325 4983 tablet_replica.cc:330] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b: stopping tablet replica
I20260927 00:44:40.000620 4983 raft_consensus.cc:2229] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:40.000825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4548
I20260927 00:44:40.000932 4983 raft_consensus.cc:2258] T 12cfd4672f4b49afb26892af5b8cdd05 P 9193bdd485ae4528ae0d027393424a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:40.001504 4981 tablet_replica.cc:330] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: stopping tablet replica
I20260927 00:44:40.001745 4981 raft_consensus.cc:2229] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:40.002048 4981 raft_consensus.cc:2258] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:40.002384 4982 tablet_replica.cc:330] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: stopping tablet replica
I20260927 00:44:40.002642 4982 raft_consensus.cc:2229] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:40.003000 4982 raft_consensus.cc:2258] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:40.005290 4982 ts_tablet_manager.cc:1905] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:40.005407 4981 ts_tablet_manager.cc:1905] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:44:40.008095 4473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44085 (error 108)
I20260927 00:44:40.008105 4981 ts_tablet_manager.cc:1918] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:40.008422 4981 log.cc:1196] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/12cfd4672f4b49afb26892af5b8cdd05
I20260927 00:44:40.008754 4981 ts_tablet_manager.cc:1939] T 12cfd4672f4b49afb26892af5b8cdd05 P ed897cb4e49e4b9e8516c44bb66d4f60: Deleting consensus metadata
I20260927 00:44:40.008773 4982 ts_tablet_manager.cc:1918] T 12cfd4672f4b49afb26892af5b8cdd05 P 06fa8cca78954d238714f18f05f83a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:40.008898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4681
I20260927 00:44:40.011126 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.010757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:52420)
0927 00:44:40.010950 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.011021 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 00:44:40.011277 4470 catalog_manager.cc:4841] TS ed897cb4e49e4b9e8516c44bb66d4f60 (127.0.98.3:41537): tablet 12cfd4672f4b49afb26892af5b8cdd05 (table test-table [id=44c186f37eba4f2c94a84ae468115ea0]) successfully deleted
W20260927 00:44:40.011309 4473 catalog_manager.cc:4584] TS 9193bdd485ae4528ae0d027393424a5b (127.0.98.1:44085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12cfd4672f4b49afb26892af5b8cdd05: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
I20260927 00:44:40.015725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4814
I20260927 00:44:40.016675 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.015757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:55252)
0927 00:44:40.016522 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.016587 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663}
I20260927 00:44:40.023505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4455
2026-09-27T00:44:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2317 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:40Z Disabled control of system clock
I20260927 00:44:40.053292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:44:40.164789 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:40.165094 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:40.165133 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:40.169409 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:40.169484 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:40.169500 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:40.169517 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:40.169533 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:40.174778 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:40.175956 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:40.177434 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:40.186121 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:40.186205 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.186190 4988 server_base.cc:1034] running on GCE node
W20260927 00:44:40.186126 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.186993 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:40.188052 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:40.189224 4988 hybrid_clock.cc:648] HybridClock initialized: now 1790469880189193 us; error 40 us; skew 500 ppm
I20260927 00:44:40.191118 4988 webserver.cc:460] Webserver started at http://127.0.98.62:39645/ using document root <none> and password file <none>
I20260927 00:44:40.191694 4988 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:40.191756 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:40.191984 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:40.194072 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "17feeffa7d6d4dbdba206ea71b010c0f"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "c5eb22d351f6dc39a83d51c96a5e522d"
server_key_iv: "9f4ef8d63df3da32c4a7d9fa5ebb8b8d"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.194657 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "17feeffa7d6d4dbdba206ea71b010c0f"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "c5eb22d351f6dc39a83d51c96a5e522d"
server_key_iv: "9f4ef8d63df3da32c4a7d9fa5ebb8b8d"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.198226 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:44:40.200536 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.201675 4988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:40.201825 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "17feeffa7d6d4dbdba206ea71b010c0f"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "c5eb22d351f6dc39a83d51c96a5e522d"
server_key_iv: "9f4ef8d63df3da32c4a7d9fa5ebb8b8d"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.201924 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:40.206267 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:40.206831 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:40.207015 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:40.213496 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45721
I20260927 00:44:40.213552 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45721 every 8 connection(s)
I20260927 00:44:40.214464 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:40.217559 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:40.218899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 4988
I20260927 00:44:40.219019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:40.219336 392 external_mini_cluster.cc:1428] Setting key efc108f97bdcf61382177be340747807
I20260927 00:44:40.224171 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: Bootstrap starting.
I20260927 00:44:40.224766 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.220694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.0.98.62:45721)
0927 00:44:40.221274 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.221290 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.221323 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:44:40.222008 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.222025 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.222369 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.222639 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.223447 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.223451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.223462 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.223471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.223521 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.223538 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.223647 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.223745 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.223991 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260927 00:44:40.226845 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:40.227596 5055 log.cc:824] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:40.229565 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: No bootstrap required, opened a new log
I20260927 00:44:40.234009 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 00:44:40.234334 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:40.234401 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17feeffa7d6d4dbdba206ea71b010c0f, State: Initialized, Role: FOLLOWER
I20260927 00:44:40.234925 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 00:44:40.235116 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:40.235193 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:40.235383 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:40.236899 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 00:44:40.237300 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17feeffa7d6d4dbdba206ea71b010c0f; no voters:
I20260927 00:44:40.237942 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:40.238108 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:40.238360 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [term 1 LEADER]: Becoming Leader. State: Replica: 17feeffa7d6d4dbdba206ea71b010c0f, State: Running, Role: LEADER
I20260927 00:44:40.238734 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 00:44:40.239504 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:40.240562 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17feeffa7d6d4dbdba206ea71b010c0f. Latest consensus state: current_term: 1 leader_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } } }
I20260927 00:44:40.240613 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17feeffa7d6d4dbdba206ea71b010c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } } }
I20260927 00:44:40.240685 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:40.240685 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:40.241118 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:40.244395 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:40.250032 5069 catalog_manager.cc:1347] Generated new cluster ID: 8c09710c32504ca7b9a761f0a7aba22e
I20260927 00:44:40.250115 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:40.264810 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:40.265163 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:40.273375 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: Generated new TSK 0
I20260927 00:44:40.274107 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:44:40.277473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:44:40.389365 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:40.389616 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:40.389657 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:40.393723 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:40.393931 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:40.394043 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:40.398972 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:40.400069 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:40.401446 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:40.411396 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:40.411437 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:40.411420 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.411660 5079 server_base.cc:1034] running on GCE node
I20260927 00:44:40.412339 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:40.413079 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:40.414252 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790469880414233 us; error 32 us; skew 500 ppm
I20260927 00:44:40.416558 5079 webserver.cc:460] Webserver started at http://127.0.98.1:33423/ using document root <none> and password file <none>
I20260927 00:44:40.417268 5079 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:40.417359 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:40.417608 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:40.419823 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ff693071c7f48a2a521f22340f6ba3c"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "4342d12381c8cde6b92bf3bbbc73b67a"
server_key_iv: "9a00b00000bd1fd8e5ff83993eee4156"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.420351 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ff693071c7f48a2a521f22340f6ba3c"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "4342d12381c8cde6b92bf3bbbc73b67a"
server_key_iv: "9a00b00000bd1fd8e5ff83993eee4156"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.424698 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:44:40.427079 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.428287 5079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:40.428457 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "1ff693071c7f48a2a521f22340f6ba3c"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "4342d12381c8cde6b92bf3bbbc73b67a"
server_key_iv: "9a00b00000bd1fd8e5ff83993eee4156"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.428586 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:40.433178 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:40.433810 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:40.434000 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:40.434525 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:40.435405 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:40.435482 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.435547 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:40.435606 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.445204 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43215
I20260927 00:44:40.445261 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43215 every 8 connection(s)
I20260927 00:44:40.446228 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:40.452319 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.448008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48995 (local address 127.0.98.62:45721)
0927 00:44:40.448164 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.448170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.449272 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 00:44:40.450571 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.450574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.450610 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.450693 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.451826 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.451852 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.451854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.451867 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.451886 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.451895 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.452026 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.452174 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.452215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:44:40.453131 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.448482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.1:48995)
0927 00:44:40.449114 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.449153 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.450381 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.450839 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.450851 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.450941 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.451559 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.451560 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.451600 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.451666 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.452017 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.452023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.452036 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.452158 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.452340 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260927 00:44:40.453871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5079
I20260927 00:44:40.453992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:40.454289 392 external_mini_cluster.cc:1428] Setting key 6968fb09abe2e7cc9301d99196599c50
I20260927 00:44:40.454511 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 00:44:40.454741 5207 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:40.455219 5207 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
I20260927 00:44:40.456252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:44:40.456753 5019 ts_manager.cc:194] Registered new tserver with Master: 1ff693071c7f48a2a521f22340f6ba3c (127.0.98.1:43215)
W20260927 00:44:40.565886 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:40.566138 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:40.566176 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:40.570004 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:40.570060 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:40.570112 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:40.574720 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:40.575776 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:40.577124 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:40.586573 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.586627 5210 server_base.cc:1034] running on GCE node
W20260927 00:44:40.586675 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:40.586591 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.587206 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:40.587842 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:40.589057 5210 hybrid_clock.cc:648] HybridClock initialized: now 1790469880589021 us; error 46 us; skew 500 ppm
I20260927 00:44:40.591048 5210 webserver.cc:460] Webserver started at http://127.0.98.2:41975/ using document root <none> and password file <none>
I20260927 00:44:40.591677 5210 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:40.591768 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:40.591979 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:40.594059 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "4df730eb4015496ba8ab25ea71102fa9"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "732baebd68eca0960e89ab9cd49c83ef"
server_key_iv: "36c1091c126e32bbf914f518f67cbde8"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.594632 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4df730eb4015496ba8ab25ea71102fa9"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "732baebd68eca0960e89ab9cd49c83ef"
server_key_iv: "36c1091c126e32bbf914f518f67cbde8"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.598268 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:44:40.600680 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.601992 5210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:40.602115 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "4df730eb4015496ba8ab25ea71102fa9"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "732baebd68eca0960e89ab9cd49c83ef"
server_key_iv: "36c1091c126e32bbf914f518f67cbde8"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.602224 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:40.606932 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:40.607583 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:40.607774 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:40.608333 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:40.609241 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:40.609310 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.609368 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:40.609421 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.618367 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45387
I20260927 00:44:40.618396 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45387 every 8 connection(s)
I20260927 00:44:40.619272 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:40.622231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5210
I20260927 00:44:40.622354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:40.622661 392 external_mini_cluster.cc:1428] Setting key 5901849742c68abc24a381b6feb6a9c5
I20260927 00:44:40.624677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:44:40.625890 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.621156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51079 (local address 127.0.98.62:45721)
0927 00:44:40.621318 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.621323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.622527 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 00:44:40.624131 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.624134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.624172 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.624251 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.625389 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.625393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.625396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.625408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.625429 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.625441 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.625539 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.625696 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.625739 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:40.626724 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.621526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.2:51079)
0927 00:44:40.622359 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.622416 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.623897 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.624436 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.624443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.624537 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.625140 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.625142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.625181 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.625247 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.625535 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.625543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.625551 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.625729 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.625983 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":112,"threads_started":1}
I20260927 00:44:40.627868 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 00:44:40.628093 5338 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:40.628472 5338 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
I20260927 00:44:40.629455 5019 ts_manager.cc:194] Registered new tserver with Master: 4df730eb4015496ba8ab25ea71102fa9 (127.0.98.2:45387)
W20260927 00:44:40.736285 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:40.736593 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:40.736667 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:40.740945 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:40.741029 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:40.741127 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:40.746023 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:40.747229 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:40.748656 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:40.758087 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.758169 5341 server_base.cc:1034] running on GCE node
W20260927 00:44:40.758095 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:40.758085 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:40.758813 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:40.759426 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:40.760608 5341 hybrid_clock.cc:648] HybridClock initialized: now 1790469880760576 us; error 43 us; skew 500 ppm
I20260927 00:44:40.762363 5341 webserver.cc:460] Webserver started at http://127.0.98.3:35467/ using document root <none> and password file <none>
I20260927 00:44:40.762912 5341 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:40.763000 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:40.763226 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:40.765161 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "a93c55e0759b49488fd1200d447a1b02"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "823f85ca2df3b9f0d34fa72806ada120"
server_key_iv: "c16ee8d6cd2a306b9a55404856374c99"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.765707 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a93c55e0759b49488fd1200d447a1b02"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "823f85ca2df3b9f0d34fa72806ada120"
server_key_iv: "c16ee8d6cd2a306b9a55404856374c99"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.769131 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:40.771682 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.773280 5341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:44:40.773401 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "a93c55e0759b49488fd1200d447a1b02"
format_stamp: "Formatted at 2026-09-27 00:44:40 on dist-test-slave-l5sb"
server_key: "823f85ca2df3b9f0d34fa72806ada120"
server_key_iv: "c16ee8d6cd2a306b9a55404856374c99"
server_key_version: "encryptionkey@0"
I20260927 00:44:40.773530 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:40.779404 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:40.780118 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:40.780292 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:40.780946 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:40.781810 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:40.781868 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.781906 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:40.781927 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:40.791138 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37905
I20260927 00:44:40.791169 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37905 every 8 connection(s)
I20260927 00:44:40.792009 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:40.798074 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.793813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53489 (local address 127.0.98.62:45721)
0927 00:44:40.793947 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.793951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.795012 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 00:44:40.796327 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.796331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.796375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.796484 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.797592 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.797596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.797598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.797610 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.797631 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.797647 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.797741 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.797891 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.797941 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:44:40.798905 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.794206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.3:53489)
0927 00:44:40.794858 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.794895 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.796113 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.796647 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.796653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.796743 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.797332 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.797334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.797372 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.797445 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.797734 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.797742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.797749 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.797936 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.798172 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260927 00:44:40.799949 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 00:44:40.800148 5469 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:40.800518 5469 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
I20260927 00:44:40.801188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5341
I20260927 00:44:40.801329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:40.801522 5019 ts_manager.cc:194] Registered new tserver with Master: a93c55e0759b49488fd1200d447a1b02 (127.0.98.3:37905)
I20260927 00:44:40.801594 392 external_mini_cluster.cc:1428] Setting key a815afe007d993daf9658d022c878b0a
I20260927 00:44:40.803447 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:40.806978 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.805269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.98.62:45721)
0927 00:44:40.805399 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.805403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.805696 (+ 293us) server_negotiation.cc:407] Connection header received
0927 00:44:40.805747 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.805749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.805843 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.805929 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.806647 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.806650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.806652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.806661 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.806677 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.806685 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.806756 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.806792 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.806821 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:44:40.809459 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:40.812374 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:40.823685 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.820379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45387 (local address 127.0.0.1:36630)
0927 00:44:40.820718 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.820770 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.820868 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.821932 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.821938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.822013 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.822546 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.822550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.822574 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.822644 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.823358 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.823367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.823373 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.823503 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.823539 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260927 00:44:40.823750 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.820517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.0.98.2:45387)
0927 00:44:40.821427 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.821439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.821468 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:44:40.821517 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.821537 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.821743 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.822272 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.823154 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.823161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.823176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.823189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.823226 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.823257 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.823374 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.823528 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.823596 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":115,"threads_started":1}
I20260927 00:44:40.824023 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.821132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.98.1:43215)
0927 00:44:40.821608 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.821620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.821643 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:40.821696 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.821703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.821913 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.822321 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.823189 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.823195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.823208 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.823217 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.823260 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.823285 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.823761 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.823803 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.823864 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":91,"threads_started":1}
I20260927 00:44:40.824034 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.821443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.0.98.3:37905)
0927 00:44:40.821765 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.821818 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.821840 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:44:40.821892 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.821897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.822068 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.822252 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.823398 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.823403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.823413 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.823426 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.823467 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.823496 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.823758 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.823803 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.823864 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
I20260927 00:44:40.823696 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.820758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.0.1:60044)
0927 00:44:40.821076 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.821088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.821158 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.822059 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.822062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.822077 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.822718 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.822719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.822730 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.822754 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.823385 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.823388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.823391 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.823462 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.823546 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":86,"threads_started":1}
I20260927 00:44:40.824930 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.821059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37905 (local address 127.0.0.1:35584)
0927 00:44:40.821267 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.821278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.821339 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.822711 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.822714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.822728 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.823227 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.823229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.823237 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.823261 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.823590 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.823593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.823597 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.823646 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.823672 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260927 00:44:40.826782 5272 tablet_service.cc:1467] Processing CreateTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 (DEFAULT_TABLE table=test-table [id=41fee092da664ad78a469304d75eefae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:40.827831 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c86ec77c464f3db90a732eb5cd9b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:40.827498 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 (DEFAULT_TABLE table=test-table [id=41fee092da664ad78a469304d75eefae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:40.827498 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 (DEFAULT_TABLE table=test-table [id=41fee092da664ad78a469304d75eefae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:40.828461 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c86ec77c464f3db90a732eb5cd9b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:40.828461 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c86ec77c464f3db90a732eb5cd9b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:40.834663 5486 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Bootstrap starting.
I20260927 00:44:40.835744 5488 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Bootstrap starting.
I20260927 00:44:40.835744 5487 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: Bootstrap starting.
I20260927 00:44:40.836719 5486 tablet_bootstrap.cc:654] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:40.837410 5486 log.cc:824] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:40.837795 5488 tablet_bootstrap.cc:654] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:40.837800 5487 tablet_bootstrap.cc:654] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:40.838618 5487 log.cc:824] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:40.838618 5488 log.cc:824] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:40.839780 5486 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: No bootstrap required, opened a new log
I20260927 00:44:40.839967 5486 ts_tablet_manager.cc:1397] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:44:40.840420 5487 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: No bootstrap required, opened a new log
I20260927 00:44:40.840514 5488 tablet_bootstrap.cc:492] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: No bootstrap required, opened a new log
I20260927 00:44:40.840603 5487 ts_tablet_manager.cc:1397] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:40.840695 5488 ts_tablet_manager.cc:1397] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:40.844312 5487 raft_consensus.cc:348] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.844563 5487 raft_consensus.cc:374] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:40.844620 5487 raft_consensus.cc:729] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff693071c7f48a2a521f22340f6ba3c, State: Initialized, Role: FOLLOWER
I20260927 00:44:40.844551 5486 raft_consensus.cc:348] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.844844 5486 raft_consensus.cc:374] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:40.844908 5486 raft_consensus.cc:729] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df730eb4015496ba8ab25ea71102fa9, State: Initialized, Role: FOLLOWER
I20260927 00:44:40.845095 5487 consensus_queue.cc:260] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.845063 5488 raft_consensus.cc:348] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.845281 5488 raft_consensus.cc:374] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:40.845325 5488 raft_consensus.cc:729] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93c55e0759b49488fd1200d447a1b02, State: Initialized, Role: FOLLOWER
I20260927 00:44:40.845381 5486 consensus_queue.cc:260] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.845811 5207 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
I20260927 00:44:40.845714 5488 consensus_queue.cc:260] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.846196 5338 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
I20260927 00:44:40.846349 5486 ts_tablet_manager.cc:1428] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:44:40.846598 5488 ts_tablet_manager.cc:1428] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:44:40.846889 5469 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
I20260927 00:44:40.848063 5487 ts_tablet_manager.cc:1428] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:44:40.855624 5493 raft_consensus.cc:482] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:40.855759 5493 raft_consensus.cc:504] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.856633 5493 leader_election.cc:288] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a93c55e0759b49488fd1200d447a1b02 (127.0.98.3:37905), 1ff693071c7f48a2a521f22340f6ba3c (127.0.98.1:43215)
I20260927 00:44:40.858530 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.856855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.98.2:42591)
0927 00:44:40.857038 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.857054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.857144 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.857369 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.857374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.857388 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.857957 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.857959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.857968 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.857990 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.858226 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.858232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.858236 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.858320 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.858348 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 00:44:40.858511 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.856928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42591 (local address 127.0.98.1:43215)
0927 00:44:40.857042 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.857048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.857107 (+ 59us) server_negotiation.cc:407] Connection header received
0927 00:44:40.857215 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.857218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.857259 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.857339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.858087 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.858092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.858095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.858107 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.858128 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.858138 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.858218 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.858324 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.858367 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:44:40.858667 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.856928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45401 (local address 127.0.98.3:37905)
0927 00:44:40.857049 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.857054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.857333 (+ 279us) server_negotiation.cc:407] Connection header received
0927 00:44:40.857397 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.857401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.857433 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.857515 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.858300 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.858305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.858310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.858321 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.858340 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.858350 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.858430 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.858513 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.858554 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:44:40.858685 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.856855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37905 (local address 127.0.98.2:45401)
0927 00:44:40.857216 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.857229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:40.857322 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:40.857542 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:40.857544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:40.857555 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:40.858143 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:44:40.858144 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:40.858151 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:44:40.858169 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:40.858434 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:40.858437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:40.858440 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:44:40.858514 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:44:40.858543 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":33,"thread_start_us":111,"threads_started":1}
I20260927 00:44:40.859231 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c86ec77c464f3db90a732eb5cd9b96" candidate_uuid: "4df730eb4015496ba8ab25ea71102fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93c55e0759b49488fd1200d447a1b02" is_pre_election: true
I20260927 00:44:40.859341 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c86ec77c464f3db90a732eb5cd9b96" candidate_uuid: "4df730eb4015496ba8ab25ea71102fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff693071c7f48a2a521f22340f6ba3c" is_pre_election: true
I20260927 00:44:40.859573 5423 raft_consensus.cc:2454] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df730eb4015496ba8ab25ea71102fa9 in term 0.
I20260927 00:44:40.859591 5161 raft_consensus.cc:2454] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df730eb4015496ba8ab25ea71102fa9 in term 0.
I20260927 00:44:40.859998 5225 leader_election.cc:302] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df730eb4015496ba8ab25ea71102fa9, a93c55e0759b49488fd1200d447a1b02; no voters:
I20260927 00:44:40.860257 5493 raft_consensus.cc:2781] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:40.860335 5493 raft_consensus.cc:482] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:40.860376 5493 raft_consensus.cc:3037] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:40.861335 5493 raft_consensus.cc:504] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.861660 5493 leader_election.cc:288] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [CANDIDATE]: Term 1 election: Requested vote from peers a93c55e0759b49488fd1200d447a1b02 (127.0.98.3:37905), 1ff693071c7f48a2a521f22340f6ba3c (127.0.98.1:43215)
I20260927 00:44:40.861953 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c86ec77c464f3db90a732eb5cd9b96" candidate_uuid: "4df730eb4015496ba8ab25ea71102fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93c55e0759b49488fd1200d447a1b02"
I20260927 00:44:40.861989 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c86ec77c464f3db90a732eb5cd9b96" candidate_uuid: "4df730eb4015496ba8ab25ea71102fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff693071c7f48a2a521f22340f6ba3c"
I20260927 00:44:40.862072 5423 raft_consensus.cc:3037] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:40.862112 5161 raft_consensus.cc:3037] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:40.863052 5423 raft_consensus.cc:2454] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df730eb4015496ba8ab25ea71102fa9 in term 1.
I20260927 00:44:40.863159 5161 raft_consensus.cc:2454] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df730eb4015496ba8ab25ea71102fa9 in term 1.
I20260927 00:44:40.863354 5225 leader_election.cc:302] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df730eb4015496ba8ab25ea71102fa9, a93c55e0759b49488fd1200d447a1b02; no voters:
I20260927 00:44:40.863504 5493 raft_consensus.cc:2781] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:40.863736 5493 raft_consensus.cc:686] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 4df730eb4015496ba8ab25ea71102fa9, State: Running, Role: LEADER
I20260927 00:44:40.864043 5493 consensus_queue.cc:237] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } }
I20260927 00:44:40.866936 5019 catalog_manager.cc:5495] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df730eb4015496ba8ab25ea71102fa9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4df730eb4015496ba8ab25ea71102fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df730eb4015496ba8ab25ea71102fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } health_report { overall_health: UNKNOWN } } }
W20260927 00:44:40.870191 5339 tablet.cc:2369] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:40.875975 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.874619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.0.98.2:45387)
0927 00:44:40.874749 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:40.874753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:40.874766 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:40.874892 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:40.874895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:40.874947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:40.875020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:40.875622 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:40.875624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:40.875626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:40.875636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:40.875654 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:40.875661 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:40.875732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:40.875786 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:44:40.875817 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:44:40.881667 5161 raft_consensus.cc:1261] T 58c86ec77c464f3db90a732eb5cd9b96 P 1ff693071c7f48a2a521f22340f6ba3c [term 1 FOLLOWER]: Refusing update from remote peer 4df730eb4015496ba8ab25ea71102fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:40.881666 5423 raft_consensus.cc:1261] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 1 FOLLOWER]: Refusing update from remote peer 4df730eb4015496ba8ab25ea71102fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:40.882413 5496 consensus_queue.cc:1035] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93c55e0759b49488fd1200d447a1b02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:40.882633 5493 consensus_queue.cc:1035] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff693071c7f48a2a521f22340f6ba3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:40.890292 5499 mvcc.cc:205] Tried to move back new op lower bound from 7333764632084439040 to 7333764632021450752. Current Snapshot: MvccSnapshot[applied={T|T < 7333764632084439040}]
I20260927 00:44:40.892578 5501 mvcc.cc:205] Tried to move back new op lower bound from 7333764632084439040 to 7333764632021450752. Current Snapshot: MvccSnapshot[applied={T|T < 7333764632084439040}]
I20260927 00:44:40.899611 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:40.899791 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:40.902058 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17feeffa7d6d4dbdba206ea71b010c0f: Sending DeleteTablet for 3 replicas of tablet 58c86ec77c464f3db90a732eb5cd9b96
I20260927 00:44:40.902817 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:40 UTC) from {username='slave'} at 127.0.0.1:60044
I20260927 00:44:40.902964 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:40 UTC) from {username='slave'} at 127.0.0.1:35584
I20260927 00:44:40.903028 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c86ec77c464f3db90a732eb5cd9b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:40 UTC) from {username='slave'} at 127.0.0.1:36630
I20260927 00:44:40.903378 5510 tablet_replica.cc:330] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: stopping tablet replica
I20260927 00:44:40.903542 5511 tablet_replica.cc:330] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: stopping tablet replica
I20260927 00:44:40.903669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5079
I20260927 00:44:40.903853 5511 raft_consensus.cc:2229] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:40.904403 5511 raft_consensus.cc:2258] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:40.906316 5511 ts_tablet_manager.cc:1905] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:40.906344 5510 raft_consensus.cc:2229] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:40.906756 5510 raft_consensus.cc:2258] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:40.908239 5510 ts_tablet_manager.cc:1905] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:40.909808 5511 ts_tablet_manager.cc:1918] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:40.909912 5511 log.cc:1196] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/58c86ec77c464f3db90a732eb5cd9b96
I20260927 00:44:40.910264 5511 ts_tablet_manager.cc:1939] T 58c86ec77c464f3db90a732eb5cd9b96 P 4df730eb4015496ba8ab25ea71102fa9: Deleting consensus metadata
I20260927 00:44:40.911036 5510 ts_tablet_manager.cc:1918] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:40.911198 5510 log.cc:1196] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/58c86ec77c464f3db90a732eb5cd9b96
W20260927 00:44:40.911413 5006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43215 (error 108)
I20260927 00:44:40.911451 5003 catalog_manager.cc:4841] TS 4df730eb4015496ba8ab25ea71102fa9 (127.0.98.2:45387): tablet 58c86ec77c464f3db90a732eb5cd9b96 (table test-table [id=41fee092da664ad78a469304d75eefae]) successfully deleted
I20260927 00:44:40.911909 5510 ts_tablet_manager.cc:1939] T 58c86ec77c464f3db90a732eb5cd9b96 P a93c55e0759b49488fd1200d447a1b02: Deleting consensus metadata
I20260927 00:44:40.912221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5210
I20260927 00:44:40.912956 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:40.912187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.0.1:60056)
0927 00:44:40.912779 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:40.912856 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
I20260927 00:44:40.913341 5003 catalog_manager.cc:4841] TS a93c55e0759b49488fd1200d447a1b02 (127.0.98.3:37905): tablet 58c86ec77c464f3db90a732eb5cd9b96 (table test-table [id=41fee092da664ad78a469304d75eefae]) successfully deleted
W20260927 00:44:40.913833 5006 catalog_manager.cc:4584] TS 1ff693071c7f48a2a521f22340f6ba3c (127.0.98.1:43215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58c86ec77c464f3db90a732eb5cd9b96: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43215: connect: Connection refused (error 111)
I20260927 00:44:40.918104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5341
I20260927 00:44:40.923467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 4988
2026-09-27T00:44:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (898 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:44:40.944844 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:44:40.946614 392 webserver.cc:460] Webserver started at https://127.0.0.1:44815/ using document root <none> and password file <none>
I20260927 00:44:40.946789 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:44:40.947072 392 webserver.cc:460] Webserver started at http://127.0.0.1:44347/ using document root <none> and password file <none>
2026-09-27T00:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:40Z Disabled control of system clock
I20260927 00:44:40.956017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34463
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44815/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/testchainca.cert with env {}
W20260927 00:44:41.063897 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:41.064146 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:41.064183 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:41.068063 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:41.068110 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:41.068125 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:41.068141 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:41.068156 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:44:41.068171 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:44:41.068184 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44815/jwks.json
I20260927 00:44:41.073351 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44815/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:41.074322 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:41.075246 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:41.083297 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:41.083284 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:41.083287 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:41.083662 5518 server_base.cc:1034] running on GCE node
I20260927 00:44:41.084208 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:41.085289 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:41.086483 5518 hybrid_clock.cc:648] HybridClock initialized: now 1790469881086461 us; error 34 us; skew 500 ppm
I20260927 00:44:41.088232 5518 webserver.cc:460] Webserver started at http://127.0.98.62:37151/ using document root <none> and password file <none>
I20260927 00:44:41.088810 5518 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:41.088900 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:41.089128 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:41.091212 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "d808e11c31174b20b2d3ae1843ee24bc"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "da802fa28d1fc0424edc0cbf81322f22"
server_key_iv: "5424a384cf7a8a547c3d1f495502edcc"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.091778 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "d808e11c31174b20b2d3ae1843ee24bc"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "da802fa28d1fc0424edc0cbf81322f22"
server_key_iv: "5424a384cf7a8a547c3d1f495502edcc"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.095323 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:41.097581 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:41.098802 5518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:41.098945 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "d808e11c31174b20b2d3ae1843ee24bc"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "da802fa28d1fc0424edc0cbf81322f22"
server_key_iv: "5424a384cf7a8a547c3d1f495502edcc"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.099056 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:41.110869 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:41.111532 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:41.111717 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:41.118058 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34463
I20260927 00:44:41.118075 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34463 every 8 connection(s)
I20260927 00:44:41.119010 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:41.121229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5518
I20260927 00:44:41.121356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:41.121634 392 external_mini_cluster.cc:1428] Setting key f0aa0588a735ea6864f62695ab180508
I20260927 00:44:41.122291 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:41.128172 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc: Bootstrap starting.
I20260927 00:44:41.130424 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:41.130607 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.122979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.62:34463)
0927 00:44:41.123620 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.123635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.123668 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:44:41.124303 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.124319 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.124701 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.125033 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:41.125446 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.126421 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.127340 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.127827 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.128001 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.128885 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:41.128890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:41.128902 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:41.129274 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:41.129325 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:41.129344 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:41.129458 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:41.129563 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:44:41.129873 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260927 00:44:41.131225 5585 log.cc:824] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:41.133404 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:44:41.133419 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc: No bootstrap required, opened a new log
I20260927 00:44:41.137133 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } }
I20260927 00:44:41.137396 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:41.137480 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d808e11c31174b20b2d3ae1843ee24bc, State: Initialized, Role: FOLLOWER
I20260927 00:44:41.137969 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } }
I20260927 00:44:41.138123 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:41.138211 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:41.138309 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:41.139209 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } }
I20260927 00:44:41.139552 5585 leader_election.cc:302] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d808e11c31174b20b2d3ae1843ee24bc; no voters:
I20260927 00:44:41.139850 5585 leader_election.cc:288] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:41.139976 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:41.140205 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [term 1 LEADER]: Becoming Leader. State: Replica: d808e11c31174b20b2d3ae1843ee24bc, State: Running, Role: LEADER
I20260927 00:44:41.140496 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } }
I20260927 00:44:41.141137 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:41.142242 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d808e11c31174b20b2d3ae1843ee24bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } } }
I20260927 00:44:41.142230 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d808e11c31174b20b2d3ae1843ee24bc. Latest consensus state: current_term: 1 leader_uuid: "d808e11c31174b20b2d3ae1843ee24bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808e11c31174b20b2d3ae1843ee24bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34463 } } }
I20260927 00:44:41.142376 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:41.142376 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:41.142799 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:41.145881 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:41.151711 5599 catalog_manager.cc:1347] Generated new cluster ID: 5c6664d6ae2c4c4eacd1b649c430987d
I20260927 00:44:41.151793 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:41.162787 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:41.163718 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:41.169476 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d808e11c31174b20b2d3ae1843ee24bc: Generated new TSK 0
I20260927 00:44:41.170204 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:44:41.200901 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:44:41.201025 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:44:41.201567 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.183229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.98.62:34463)
0927 00:44:41.183363 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.183368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.183557 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:44:41.183746 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.183749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.183805 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.183917 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:44:41.184148 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.184998 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.186057 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.186414 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.186516 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.186893 (+ 377us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:44:41.201166 (+ 14273us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:44:41.201427 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:44:41.201653 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:44:41.202844 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:41.183113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34463 (local address 127.0.0.1:38754)
0927 00:44:41.183474 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:41.183486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:41.183595 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:41.183936 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:41.183938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:41.183944 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:44:41.184046 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.184051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.185150 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.185153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.185937 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.185946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.186514 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.186537 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.186605 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.186692 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:44:41.201427 (+ 14735us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:44:41.201658 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:44:41.203722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5518
2026-09-27T00:44:41Z chronyd exiting
I20260927 00:44:41.241274 392 test_util.cc:175] -----------------------------------------------
I20260927 00:44:41.241405 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469854491772-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:44:41.242529 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:44:41.243520 392 webserver.cc:460] Webserver started at https://127.0.0.1:42697/ using document root <none> and password file <none>
I20260927 00:44:41.243651 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:44:41.243927 392 webserver.cc:460] Webserver started at http://127.0.0.1:41723/ using document root <none> and password file <none>
2026-09-27T00:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:41Z Disabled control of system clock
I20260927 00:44:41.252594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41991
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/testchainca.cert with env {}
W20260927 00:44:41.360167 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:41.360433 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:41.360473 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:41.364580 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:41.364638 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:41.364655 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:41.364670 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:41.364727 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:44:41.364758 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:44:41.364786 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42697/jwks.json
I20260927 00:44:41.370725 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:41.371775 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:41.372773 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:41.381198 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:41.381237 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:41.381207 5621 server_base.cc:1034] running on GCE node
W20260927 00:44:41.381227 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:41.382102 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:41.383090 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:41.384249 5621 hybrid_clock.cc:648] HybridClock initialized: now 1790469881384233 us; error 29 us; skew 500 ppm
I20260927 00:44:41.386080 5621 webserver.cc:460] Webserver started at http://127.0.98.62:32779/ using document root <none> and password file <none>
I20260927 00:44:41.386612 5621 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:41.386668 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:41.386819 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:41.388835 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "b1209818948148ab81184065f051eecf"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "dd32c22e56f4a4b5b8c2a55d18a6bc48"
server_key_iv: "00726e4c10c1fc9ea157dd5eecc42302"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.389338 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1209818948148ab81184065f051eecf"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "dd32c22e56f4a4b5b8c2a55d18a6bc48"
server_key_iv: "00726e4c10c1fc9ea157dd5eecc42302"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.392915 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:44:41.395256 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:41.396452 5621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:41.396574 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "b1209818948148ab81184065f051eecf"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "dd32c22e56f4a4b5b8c2a55d18a6bc48"
server_key_iv: "00726e4c10c1fc9ea157dd5eecc42302"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.396684 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:41.433320 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:41.434114 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:41.434314 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:41.441097 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41991
I20260927 00:44:41.441108 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41991 every 8 connection(s)
I20260927 00:44:41.442135 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:41.445384 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:41.447988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5621
I20260927 00:44:41.448127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:41.448431 392 external_mini_cluster.cc:1428] Setting key f718e8047cde8e9f92e88f77328c9662
I20260927 00:44:41.451508 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf: Bootstrap starting.
I20260927 00:44:41.453621 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:41.454383 5688 log.cc:824] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:41.456378 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf: No bootstrap required, opened a new log
I20260927 00:44:41.457825 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.449941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.0.98.62:41991)
0927 00:44:41.450572 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.450587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.450623 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:44:41.451226 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.451242 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.451694 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.452029 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:41.452468 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.453396 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.454389 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.455065 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.455226 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.456104 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:41.456109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:41.456121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:41.456531 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:41.456593 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:41.456613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:41.456715 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:41.456859 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:44:41.457107 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260927 00:44:41.460570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:44:41.460480 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } }
I20260927 00:44:41.460743 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:41.460799 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1209818948148ab81184065f051eecf, State: Initialized, Role: FOLLOWER
I20260927 00:44:41.461225 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } }
I20260927 00:44:41.461382 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:41.461450 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:41.461544 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:41.462554 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } }
I20260927 00:44:41.462870 5688 leader_election.cc:302] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1209818948148ab81184065f051eecf; no voters:
I20260927 00:44:41.463147 5688 leader_election.cc:288] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:41.463274 5693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:41.463487 5693 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [term 1 LEADER]: Becoming Leader. State: Replica: b1209818948148ab81184065f051eecf, State: Running, Role: LEADER
I20260927 00:44:41.463778 5693 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } }
I20260927 00:44:41.464282 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:41.465312 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1209818948148ab81184065f051eecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } } }
I20260927 00:44:41.465441 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:41.465807 5702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:41.465755 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1209818948148ab81184065f051eecf. Latest consensus state: current_term: 1 leader_uuid: "b1209818948148ab81184065f051eecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1209818948148ab81184065f051eecf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41991 } } }
I20260927 00:44:41.465991 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:41.468756 5702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:41.474323 5702 catalog_manager.cc:1347] Generated new cluster ID: 1ec819d34b8044c2b482018a4b4598ae
I20260927 00:44:41.474399 5702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:41.482820 5702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:41.483742 5702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:41.489794 5702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1209818948148ab81184065f051eecf: Generated new TSK 0
I20260927 00:44:41.490599 5702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:44:41.525411 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:44:41.525511 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:44:41.526216 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.502679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.0.98.62:41991)
0927 00:44:41.502817 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.502822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.503025 (+ 203us) server_negotiation.cc:407] Connection header received
0927 00:44:41.503177 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.503179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.503226 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.503316 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:44:41.503570 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.504094 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.505342 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.505572 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.505652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.505992 (+ 340us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:44:41.525846 (+ 19854us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:44:41.526098 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:44:41.526304 5691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:44:41.526422 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:41.502606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41991 (local address 127.0.0.1:50318)
0927 00:44:41.502933 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:41.502948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:41.503059 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:41.503336 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:41.503340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:41.503347 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:44:41.503463 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.503473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.504224 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.504228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.505221 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.505231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.505667 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.505685 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.505808 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.505845 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:44:41.526058 (+ 20213us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:44:41.526242 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260927 00:44:41.527100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5621
2026-09-27T00:44:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:44:41.565500 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:44:41.566411 392 webserver.cc:460] Webserver started at https://127.0.0.1:33045/ using document root <none> and password file <none>
I20260927 00:44:41.566493 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:44:41.566696 392 webserver.cc:460] Webserver started at http://127.0.0.1:46125/ using document root <none> and password file <none>
2026-09-27T00:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:41Z Disabled control of system clock
I20260927 00:44:41.575321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35507
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35507
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33045/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:44:41.684806 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:41.685068 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:41.685113 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:41.689168 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:41.689227 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:41.689244 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:41.689260 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:41.689275 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:44:41.689291 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:44:41.689304 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33045/jwks.json
I20260927 00:44:41.694125 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33045/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:41.695096 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:41.696362 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:41.704886 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:41.704913 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:41.704896 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:41.705330 5724 server_base.cc:1034] running on GCE node
I20260927 00:44:41.705989 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:41.707041 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:41.708209 5724 hybrid_clock.cc:648] HybridClock initialized: now 1790469881708180 us; error 51 us; skew 500 ppm
I20260927 00:44:41.710207 5724 webserver.cc:460] Webserver started at http://127.0.98.62:45823/ using document root <none> and password file <none>
I20260927 00:44:41.710768 5724 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:41.710830 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:41.711040 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:41.713007 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "90582efb53ff4301b6446e955f77ed29"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "ba3b4272207da8bcd574398cb837f0e5"
server_key_iv: "88c8d0ea25d395290892deedda696860"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.713562 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "90582efb53ff4301b6446e955f77ed29"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "ba3b4272207da8bcd574398cb837f0e5"
server_key_iv: "88c8d0ea25d395290892deedda696860"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.717131 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:41.719786 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:41.721165 5724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:44:41.721297 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "90582efb53ff4301b6446e955f77ed29"
format_stamp: "Formatted at 2026-09-27 00:44:41 on dist-test-slave-l5sb"
server_key: "ba3b4272207da8bcd574398cb837f0e5"
server_key_iv: "88c8d0ea25d395290892deedda696860"
server_key_version: "encryptionkey@0"
I20260927 00:44:41.721443 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:41.735369 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:41.736028 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:41.736212 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:41.742791 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35507
I20260927 00:44:41.742843 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35507 every 8 connection(s)
I20260927 00:44:41.743785 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:41.746912 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:41.750766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5724
I20260927 00:44:41.750886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:41.751173 392 external_mini_cluster.cc:1428] Setting key 901168580a578296ff5e13a6921ddacf
I20260927 00:44:41.752875 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: Bootstrap starting.
I20260927 00:44:41.755095 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:41.755812 5791 log.cc:824] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:41.757551 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: No bootstrap required, opened a new log
I20260927 00:44:41.760367 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.752518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.0.98.62:35507)
0927 00:44:41.753208 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.753226 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.753262 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:44:41.753962 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.753981 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.754369 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.754702 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:41.755168 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.756209 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.757111 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.757572 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.757837 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.758685 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:41.758690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:41.758702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:41.759084 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:41.759137 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:41.759157 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:41.759267 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:41.759421 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:44:41.759663 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260927 00:44:41.761312 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } }
I20260927 00:44:41.761583 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:41.761631 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90582efb53ff4301b6446e955f77ed29, State: Initialized, Role: FOLLOWER
I20260927 00:44:41.762084 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } }
I20260927 00:44:41.762270 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:41.762337 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:41.762432 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:41.763264 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:44:41.763584 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } }
I20260927 00:44:41.763906 5791 leader_election.cc:302] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90582efb53ff4301b6446e955f77ed29; no voters:
I20260927 00:44:41.764211 5791 leader_election.cc:288] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:41.764317 5796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:41.764536 5796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [term 1 LEADER]: Becoming Leader. State: Replica: 90582efb53ff4301b6446e955f77ed29, State: Running, Role: LEADER
I20260927 00:44:41.764846 5796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } }
I20260927 00:44:41.765488 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:41.765895 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90582efb53ff4301b6446e955f77ed29. Latest consensus state: current_term: 1 leader_uuid: "90582efb53ff4301b6446e955f77ed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } } }
I20260927 00:44:41.766026 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:41.766304 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90582efb53ff4301b6446e955f77ed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90582efb53ff4301b6446e955f77ed29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35507 } } }
I20260927 00:44:41.766428 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29 [sys.catalog]: This master's current role is: LEADER
W20260927 00:44:41.770364 5812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:41.770447 5812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:41.770520 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:41.771271 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:41.776321 5800 catalog_manager.cc:1347] Generated new cluster ID: 0ac24eeac1e148989d0c4cf221f38702
I20260927 00:44:41.776428 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:41.790138 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:41.791040 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:41.803201 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90582efb53ff4301b6446e955f77ed29: Generated new TSK 0
I20260927 00:44:41.803907 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:44:41.820902 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:41.816254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.0.98.62:35507)
0927 00:44:41.816385 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:41.816392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:41.816563 (+ 171us) server_negotiation.cc:407] Connection header received
0927 00:44:41.816762 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:41.816765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:41.816831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:41.816975 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:44:41.817217 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.817751 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.818681 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.818918 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.819027 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.819336 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:44:41.820503 (+ 1167us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:44:41.820740 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:44:41.820981 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:41.816171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35507 (local address 127.0.0.1:47312)
0927 00:44:41.816458 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:41.816474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:41.816599 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:41.816973 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:41.816977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:41.816986 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:44:41.817101 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.817107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.817915 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.817921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.818561 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:41.818568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:41.819018 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:41.819041 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:41.819128 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:41.819159 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:44:41.820670 (+ 1511us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:44:41.820809 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20260927 00:44:41.820982 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 00:44:41.821671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5724
E20260927 00:44:41.832022 5821 webserver.cc:533] Webserver: sslize failed:
E20260927 00:44:41.832090 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:44:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:41Z Disabled control of system clock
I20260927 00:44:41.873345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42695
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/testchainca.cert with env {}
W20260927 00:44:41.997118 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:41.997411 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:41.997455 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:42.001499 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:42.001561 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:42.001576 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:42.001593 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:42.001608 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:44:42.001623 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:44:42.001637 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:44:42.007004 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:42.008028 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:42.009006 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:42.017642 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:42.017614 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:42.017639 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:42.017805 5825 server_base.cc:1034] running on GCE node
I20260927 00:44:42.018385 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:42.019546 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:42.020702 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790469882020671 us; error 42 us; skew 500 ppm
I20260927 00:44:42.022572 5825 webserver.cc:460] Webserver started at http://127.0.98.62:39581/ using document root <none> and password file <none>
I20260927 00:44:42.023144 5825 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:42.023207 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:42.023447 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:42.025514 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "38653fc20d144a0ebdbc64e76607cd0a"
format_stamp: "Formatted at 2026-09-27 00:44:42 on dist-test-slave-l5sb"
server_key: "2a7b01e6a199c54516790b0abaf57063"
server_key_iv: "0c3f736a460e39d37f55d4137834d9a4"
server_key_version: "encryptionkey@0"
I20260927 00:44:42.026129 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "38653fc20d144a0ebdbc64e76607cd0a"
format_stamp: "Formatted at 2026-09-27 00:44:42 on dist-test-slave-l5sb"
server_key: "2a7b01e6a199c54516790b0abaf57063"
server_key_iv: "0c3f736a460e39d37f55d4137834d9a4"
server_key_version: "encryptionkey@0"
I20260927 00:44:42.029695 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:44:42.032179 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:42.033452 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:42.033584 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "38653fc20d144a0ebdbc64e76607cd0a"
format_stamp: "Formatted at 2026-09-27 00:44:42 on dist-test-slave-l5sb"
server_key: "2a7b01e6a199c54516790b0abaf57063"
server_key_iv: "0c3f736a460e39d37f55d4137834d9a4"
server_key_version: "encryptionkey@0"
I20260927 00:44:42.033692 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:42.046077 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:42.046706 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:42.046897 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:42.053751 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42695
I20260927 00:44:42.053759 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42695 every 8 connection(s)
I20260927 00:44:42.054726 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:42.057751 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:42.059937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5825
I20260927 00:44:42.060066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:42.060348 392 external_mini_cluster.cc:1428] Setting key 00512bcc8bb3ef6f3c53212090df5a49
I20260927 00:44:42.064078 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a: Bootstrap starting.
I20260927 00:44:42.066612 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:42.067519 5892 log.cc:824] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:42.069419 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:42.061657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.98.62:42695)
0927 00:44:42.062362 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:42.062378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:42.062414 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:44:42.063084 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:42.063100 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:42.063487 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:42.063763 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:42.064092 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.065090 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.066282 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.066798 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.066922 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:42.067771 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:42.067776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:42.067788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:44:42.068163 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:42.068217 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:44:42.068236 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:42.068309 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:42.068450 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:44:42.068753 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":182,"threads_started":1}
I20260927 00:44:42.069447 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a: No bootstrap required, opened a new log
I20260927 00:44:42.072355 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:44:42.072597 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } }
I20260927 00:44:42.072831 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:42.072896 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38653fc20d144a0ebdbc64e76607cd0a, State: Initialized, Role: FOLLOWER
I20260927 00:44:42.073359 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } }
I20260927 00:44:42.073522 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:42.073587 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:42.073680 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:42.074671 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } }
I20260927 00:44:42.075016 5892 leader_election.cc:302] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38653fc20d144a0ebdbc64e76607cd0a; no voters:
I20260927 00:44:42.075341 5892 leader_election.cc:288] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:42.075420 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:42.075690 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [term 1 LEADER]: Becoming Leader. State: Replica: 38653fc20d144a0ebdbc64e76607cd0a, State: Running, Role: LEADER
I20260927 00:44:42.075984 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } }
I20260927 00:44:42.076598 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:42.077858 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38653fc20d144a0ebdbc64e76607cd0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } } }
I20260927 00:44:42.077858 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38653fc20d144a0ebdbc64e76607cd0a. Latest consensus state: current_term: 1 leader_uuid: "38653fc20d144a0ebdbc64e76607cd0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38653fc20d144a0ebdbc64e76607cd0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42695 } } }
I20260927 00:44:42.078022 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:42.078022 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:42.078423 5907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:42.081600 5907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:42.087493 5907 catalog_manager.cc:1347] Generated new cluster ID: 421566ce34fb46138e58889e99bb8980
I20260927 00:44:42.087566 5907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:42.105650 5907 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:42.106637 5907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:42.118921 5907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38653fc20d144a0ebdbc64e76607cd0a: Generated new TSK 0
I20260927 00:44:42.119671 5907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:44:42.193043 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:42.131254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.98.62:42695)
0927 00:44:42.131389 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:42.131394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:42.131656 (+ 262us) server_negotiation.cc:407] Connection header received
0927 00:44:42.131799 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:42.131803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:42.131857 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:42.131974 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:44:42.132182 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.133015 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.133898 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.134244 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.134344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:42.134620 (+ 276us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:44:42.192551 (+ 57931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:44:42.192892 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57556: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:44:42.193138 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57556: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 00:44:42.193192 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:42.131209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42695 (local address 127.0.0.1:57556)
0927 00:44:42.131568 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:42.131580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:42.131671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:42.131988 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:42.131990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:42.131996 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:44:42.132085 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:42.132090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.133125 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.133127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:42.133749 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:42.133756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:42.134342 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:42.134360 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:42.134430 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:42.134460 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:44:42.192801 (+ 58341us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:44:42.192999 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260927 00:44:42.194108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5825
2026-09-27T00:44:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (352 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:42 dist-test-slave-l5sb krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:42 dist-test-slave-l5sb krb5kdc[5933](info): set up 2 sockets
Sep 27 00:44:42 dist-test-slave-l5sb krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260927 00:44:43.486114 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:43 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:43.604707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:43.712333 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:43.712594 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:43.712633 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:43.716635 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:43.716691 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:43.716709 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:43.716727 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:43.716769 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:43.722103 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:43.723122 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:43.724123 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:43.732326 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:43.732378 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:43.732499 5946 server_base.cc:1034] running on GCE node
W20260927 00:44:43.732355 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:43.733193 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:43.734292 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:43.735495 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790469883735477 us; error 30 us; skew 500 ppm
Sep 27 00:44:43 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:43.738201 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:43.739001 5946 webserver.cc:460] Webserver started at http://127.0.98.62:41083/ using document root <none> and password file <none>
I20260927 00:44:43.739497 5946 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:43.739552 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:43.739706 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:43.741576 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "eea862ed8cc1485b88831bdb336c8ad0"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f55f573661283bfeaf7f1b70aaa7b3ee"
server_key_iv: "8ac88f92b6738f1e6b027daf419b3b0d"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.742125 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "eea862ed8cc1485b88831bdb336c8ad0"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f55f573661283bfeaf7f1b70aaa7b3ee"
server_key_iv: "8ac88f92b6738f1e6b027daf419b3b0d"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.745502 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:43.747814 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:43.749173 5946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:43.749323 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "eea862ed8cc1485b88831bdb336c8ad0"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f55f573661283bfeaf7f1b70aaa7b3ee"
server_key_iv: "8ac88f92b6738f1e6b027daf419b3b0d"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.749432 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:43.758085 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:43.760742 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:43.760929 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:43.767453 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45949
I20260927 00:44:43.767783 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45949 every 8 connection(s)
I20260927 00:44:43.768388 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:43.770421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 5946
I20260927 00:44:43.770551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:43.770843 392 external_mini_cluster.cc:1428] Setting key df757d1c4b0211d48555315a808d99c4
I20260927 00:44:43.771284 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:43.777179 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0: Bootstrap starting.
Sep 27 00:44:43 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:43.779701 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:43.780448 6014 log.cc:824] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:43.782519 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0: No bootstrap required, opened a new log
I20260927 00:44:43.784713 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:43.772159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.62:45949)
0927 00:44:43.772574 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:43.772588 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:43.772622 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:44:43.773268 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:43.773285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:43.773589 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:43.773966 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:43.774746 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:43.775724 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:43.776580 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:43.777049 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:43.777220 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:43.780061 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:43.780066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:43.780078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:43.780105 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:43.781955 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:43.782454 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:43.782460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:43.782464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:43.782599 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:43.782898 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:43.782902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:43.782903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:43.783289 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:43.783502 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:43.783734 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:44:43.784025 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":146,"threads_started":1}
I20260927 00:44:43.786073 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } }
I20260927 00:44:43.786306 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:43.786382 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea862ed8cc1485b88831bdb336c8ad0, State: Initialized, Role: FOLLOWER
I20260927 00:44:43.786857 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } }
I20260927 00:44:43.787031 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:43.787094 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:43.787186 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:43.788049 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } }
I20260927 00:44:43.788379 6014 leader_election.cc:302] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea862ed8cc1485b88831bdb336c8ad0; no voters:
I20260927 00:44:43.788697 6014 leader_election.cc:288] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:43.788897 6019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:43.789170 6019 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [term 1 LEADER]: Becoming Leader. State: Replica: eea862ed8cc1485b88831bdb336c8ad0, State: Running, Role: LEADER
I20260927 00:44:43.789485 6019 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } }
I20260927 00:44:43.789968 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:43.791190 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea862ed8cc1485b88831bdb336c8ad0. Latest consensus state: current_term: 1 leader_uuid: "eea862ed8cc1485b88831bdb336c8ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } } }
I20260927 00:44:43.791324 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:43.791710 6028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:43.791661 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea862ed8cc1485b88831bdb336c8ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea862ed8cc1485b88831bdb336c8ad0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45949 } } }
I20260927 00:44:43.792259 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:43.794750 6028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:43.800303 6028 catalog_manager.cc:1347] Generated new cluster ID: 19949dbf6a2f4a32a0c405c9ee713296
I20260927 00:44:43.800377 6028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:43.811949 6028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:43.812984 6028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:43.817251 6028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea862ed8cc1485b88831bdb336c8ad0: Generated new TSK 0
I20260927 00:44:43.817945 6028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:43.849331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:43.960214 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:43.960512 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:43.960577 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:43.964867 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:43.964926 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:43.964985 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:43.970204 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:43.971200 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:43.972136 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:43.981355 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:43.981367 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:43.981362 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:43.981436 6040 server_base.cc:1034] running on GCE node
I20260927 00:44:43.982198 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:43.982812 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:43.984000 6040 hybrid_clock.cc:648] HybridClock initialized: now 1790469883983987 us; error 28 us; skew 500 ppm
Sep 27 00:44:43 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:43.986848 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:43.987772 6040 webserver.cc:460] Webserver started at http://127.0.98.1:39465/ using document root <none> and password file <none>
I20260927 00:44:43.988332 6040 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:43.988420 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:43.988647 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:43.990787 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "d402524d132e42c084594b67e055c0b5"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f054d8048af107e123106decb2a19700"
server_key_iv: "4bf553b840cd5bf1a1b900e8a7fe6733"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.991429 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d402524d132e42c084594b67e055c0b5"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f054d8048af107e123106decb2a19700"
server_key_iv: "4bf553b840cd5bf1a1b900e8a7fe6733"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.995103 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:43.997469 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:43.998791 6040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:43.998934 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "d402524d132e42c084594b67e055c0b5"
format_stamp: "Formatted at 2026-09-27 00:44:43 on dist-test-slave-l5sb"
server_key: "f054d8048af107e123106decb2a19700"
server_key_iv: "4bf553b840cd5bf1a1b900e8a7fe6733"
server_key_version: "encryptionkey@0"
I20260927 00:44:43.999054 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:44.032773 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:44.035993 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:44.036198 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:44.036814 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:44.037688 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:44.037763 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.037861 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:44.037914 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.047053 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36949
I20260927 00:44:44.047080 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36949 every 8 connection(s)
I20260927 00:44:44.047946 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:44.056103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6040
I20260927 00:44:44.056231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:44.056528 392 external_mini_cluster.cc:1428] Setting key da7ef22ea0db2dcb093a47c6988bbd2a
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:44.062508 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.049710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48929 (local address 127.0.98.62:45949)
0927 00:44:44.049888 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.049892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.050723 (+ 831us) server_negotiation.cc:407] Connection header received
0927 00:44:44.052124 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.052127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.052181 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.052268 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.053431 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.054032 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.055161 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.055399 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.055508 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.058363 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.058368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.058372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.058403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.060074 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.060724 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.060726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.060728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.060811 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.061259 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.061269 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.061271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.061479 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.061565 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.062238 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.062378 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:44:44.063699 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.050125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45949 (local address 127.0.98.1:48929)
0927 00:44:44.050565 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:44.050607 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:44.051917 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:44.052451 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:44.052465 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:44.052834 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:44.053294 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.053307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.054149 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.054153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.055040 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.055049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.055542 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.055546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.055714 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.056379 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:44.056396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:44.058166 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:44.060210 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.060218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.060234 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.060605 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.060911 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.060915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.060917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.061112 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.061591 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:44.061600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:44.061942 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 00:44:44.062173 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 00:44:44.062552 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260927 00:44:44.064981 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45949
I20260927 00:44:44.065220 6169 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:44.065689 6169 heartbeater.cc:511] Master 127.0.98.62:45949 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:44.067178 5978 ts_manager.cc:194] Registered new tserver with Master: d402524d132e42c084594b67e055c0b5 (127.0.98.1:36949)
I20260927 00:44:44.068374 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:44.084784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:44.193480 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:44.193720 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:44.193761 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:44.197726 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:44.197829 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:44.197891 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:44.203084 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:44.204037 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:44.204995 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:44.214105 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:44.214105 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:44.214095 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:44.214373 6174 server_base.cc:1034] running on GCE node
I20260927 00:44:44.214816 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:44.215451 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:44.216632 6174 hybrid_clock.cc:648] HybridClock initialized: now 1790469884216616 us; error 32 us; skew 500 ppm
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.219262 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:44.220134 6174 webserver.cc:460] Webserver started at http://127.0.98.2:42359/ using document root <none> and password file <none>
I20260927 00:44:44.220683 6174 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:44.220772 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:44.221014 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:44.223073 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ae31abd0fb2488a87bb00a98d51e65a"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "511ccbbc5354c2a74431c785049e2bfb"
server_key_iv: "f13ac3228a8671b696f31cfad44c85a6"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.223644 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ae31abd0fb2488a87bb00a98d51e65a"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "511ccbbc5354c2a74431c785049e2bfb"
server_key_iv: "f13ac3228a8671b696f31cfad44c85a6"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.227370 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:44.229888 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.231187 6174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:44:44.231355 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "8ae31abd0fb2488a87bb00a98d51e65a"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "511ccbbc5354c2a74431c785049e2bfb"
server_key_iv: "f13ac3228a8671b696f31cfad44c85a6"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.231474 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:44.249701 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:44.253364 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:44.253607 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:44.254330 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:44.255223 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:44.255308 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.255410 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:44.255470 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.264847 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44043
I20260927 00:44:44.264866 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44043 every 8 connection(s)
I20260927 00:44:44.265839 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:44.270975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6174
I20260927 00:44:44.271104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:44.271404 392 external_mini_cluster.cc:1428] Setting key 7b36e196797ee88d6e1bedaf2eb401d1
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:44.281177 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.267690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50183 (local address 127.0.98.62:45949)
0927 00:44:44.267825 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.267829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.268706 (+ 877us) server_negotiation.cc:407] Connection header received
0927 00:44:44.270143 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.270146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.270195 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.270283 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.271949 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.272489 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.274202 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.274425 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.274586 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.277169 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.277176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.277180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.277211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.278983 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.279544 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.279548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.279550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.279644 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.279967 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.279972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.279974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.280219 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.280334 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.280860 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.280997 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:44.282004 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.268126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45949 (local address 127.0.98.2:50183)
0927 00:44:44.268551 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:44.268592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:44.269908 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:44.270500 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:44.270508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:44.271071 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:44.271775 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.271812 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.272619 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.272622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.274060 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.274073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.274643 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.274650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.274823 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.275468 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:44.275485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:44.276995 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:44.279111 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.279116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.279129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.279436 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.279741 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.279743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.279745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.279873 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.280366 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:44.280371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:44.280601 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 00:44:44.280760 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:44:44.281015 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260927 00:44:44.283339 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45949
I20260927 00:44:44.283596 6303 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:44.284065 6303 heartbeater.cc:511] Master 127.0.98.62:45949 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:44:44.285557 5978 ts_manager.cc:194] Registered new tserver with Master: 8ae31abd0fb2488a87bb00a98d51e65a (127.0.98.2:44043)
I20260927 00:44:44.286216 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:44.299156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:44.409409 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:44.409675 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:44.409718 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:44.414216 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:44.414269 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:44.414337 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:44.420019 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:44.421002 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:44.422041 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:44.431516 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:44.431569 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:44.431643 6308 server_base.cc:1034] running on GCE node
W20260927 00:44:44.431829 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:44.432214 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:44.432821 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:44.434015 6308 hybrid_clock.cc:648] HybridClock initialized: now 1790469884434003 us; error 29 us; skew 500 ppm
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.436842 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:44.437734 6308 webserver.cc:460] Webserver started at http://127.0.98.3:37933/ using document root <none> and password file <none>
I20260927 00:44:44.438309 6308 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:44.438398 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:44.438607 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:44.440574 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbc9d8d5b3d24deba41c87aac3dadf1c"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "6cf86eab8ac806d2e0c8918ccf87207b"
server_key_iv: "f9944323984c9a1898aabe3e426cee3a"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.441119 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbc9d8d5b3d24deba41c87aac3dadf1c"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "6cf86eab8ac806d2e0c8918ccf87207b"
server_key_iv: "f9944323984c9a1898aabe3e426cee3a"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.444643 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:44:44.446960 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.448151 6308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:44.448282 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "dbc9d8d5b3d24deba41c87aac3dadf1c"
format_stamp: "Formatted at 2026-09-27 00:44:44 on dist-test-slave-l5sb"
server_key: "6cf86eab8ac806d2e0c8918ccf87207b"
server_key_iv: "f9944323984c9a1898aabe3e426cee3a"
server_key_version: "encryptionkey@0"
I20260927 00:44:44.448390 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:44.473939 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:44.476787 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:44.476996 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:44.477573 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:44.478466 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:44.478542 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.478614 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:44.478665 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:44.487457 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46873
I20260927 00:44:44.487486 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46873 every 8 connection(s)
I20260927 00:44:44.488324 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:44.495540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6308
I20260927 00:44:44.495664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:44.495959 392 external_mini_cluster.cc:1428] Setting key 46d24481a0e22cf8cae2bba6e5ad0a51
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:44.502046 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.490207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59271 (local address 127.0.98.62:45949)
0927 00:44:44.490333 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.490337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.491271 (+ 934us) server_negotiation.cc:407] Connection header received
0927 00:44:44.492722 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.492726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.492784 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.492888 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.494084 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.494614 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.495994 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.496246 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.496373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.499109 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.499112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.499114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.499136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.500353 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.500785 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.500787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.500788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.500854 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.501135 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.501137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.501138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.501292 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.501362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.501819 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.501919 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:44.502928 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.490644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45949 (local address 127.0.98.3:59271)
0927 00:44:44.491133 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:44.491176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:44.492516 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:44.493065 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:44.493072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:44.493407 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:44.493942 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.493963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.494754 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.494757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.495866 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:44.495877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.496410 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.496413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:44.496617 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.497310 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:44.497326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:44.498967 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:44.500447 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.500451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.500461 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.500705 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.500928 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:44.500930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:44.500932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:44.501063 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:44.501375 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:44.501379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:44.501590 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 00:44:44.501734 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:44:44.502030 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260927 00:44:44.503934 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45949
I20260927 00:44:44.504174 6437 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:44.504628 6437 heartbeater.cc:511] Master 127.0.98.62:45949 requested a full tablet report, sending...
I20260927 00:44:44.505558 5978 ts_manager.cc:194] Registered new tserver with Master: dbc9d8d5b3d24deba41c87aac3dadf1c (127.0.98.3:46873)
I20260927 00:44:44.506137 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59271
I20260927 00:44:44.510047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/test-admin.keytab.
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.525985 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:44:44.528594 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:44.527324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.0.98.1:36949)
0927 00:44:44.527579 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.527598 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.527620 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:44:44.527668 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.527672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.527847 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.527968 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.528373 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42624: BlockingRecv error: recv got EOF from 127.0.0.1:42624 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
W20260927 00:44:44.528653 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.529738 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.530856 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:44.540524 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.531721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.0.98.1:36949)
0927 00:44:44.531854 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.531859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.531965 (+ 106us) server_negotiation.cc:407] Connection header received
0927 00:44:44.532117 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.532119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.532161 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.532244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.532780 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.533550 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.534450 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.534897 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.535020 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.537271 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.537279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.537291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.537312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.538761 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.539192 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.539205 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.539206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.539293 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.539567 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.539570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.539572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.539798 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.540011 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.540230 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.540388 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:44:44.544111 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:44.543206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.98.1:36949)
0927 00:44:44.543325 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.543329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.543379 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:44:44.543460 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.543462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.543495 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.543558 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.544009 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42642: BlockingRecv error: recv got EOF from 127.0.0.1:42642 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:44:44.544148 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.545086 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.545979 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:44.554551 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.546748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.98.1:36949)
0927 00:44:44.546862 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.546866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.546980 (+ 114us) server_negotiation.cc:407] Connection header received
0927 00:44:44.547106 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.547109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.547150 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.547230 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.547700 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.548189 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.548952 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.549169 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.549270 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.551356 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.551359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.551364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.551391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.552952 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.553369 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.553373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.553375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.553458 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.553723 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.553726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.553728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.553929 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.554062 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.554289 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.554399 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:44:44.556851 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:44.556069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.1:36949)
0927 00:44:44.556194 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.556197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.556210 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:44.556315 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.556317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.556351 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.556415 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.556784 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42666: BlockingRecv error: recv got EOF from 127.0.0.1:42666 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:44:44.556945 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469854491772-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.557991 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:44.558847 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:44.566421 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:44.559557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.98.1:36949)
0927 00:44:44.559665 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:44.559668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:44.559782 (+ 114us) server_negotiation.cc:407] Connection header received
0927 00:44:44.559857 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:44.559859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:44.559889 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:44.559952 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:44.560373 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.560855 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.561553 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:44.561810 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:44.561903 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:44.563901 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:44.563904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:44.563906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:44.563925 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:44.565113 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.565458 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.565460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.565461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.565524 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:44.565730 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:44.565732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:44.565733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:44.565960 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:44.566053 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:44.566237 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:44:44.566313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:44.567281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6040
I20260927 00:44:44.573005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6174
I20260927 00:44:44.578528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6308
I20260927 00:44:44.583719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 5946
2026-09-27T00:44:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2143 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[6457](info): set up 2 sockets
Sep 27 00:44:44 dist-test-slave-l5sb krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260927 00:44:45.622326 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:45 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:44:45.738656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:45.847143 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:45.847400 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:45.847440 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:45.851341 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:45.851388 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:45.851408 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:45.851423 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:45.851438 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:45.856673 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:45.857643 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:45.858615 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:45.866757 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:45.866763 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:45.866778 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:45.867065 6470 server_base.cc:1034] running on GCE node
I20260927 00:44:45.867724 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:45.868734 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:45.869844 6470 hybrid_clock.cc:648] HybridClock initialized: now 1790469885869827 us; error 29 us; skew 500 ppm
Sep 27 00:44:45 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:45.875842 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:44:45.876739 6470 webserver.cc:460] Webserver started at http://127.0.98.62:37239/ using document root <none> and password file <none>
I20260927 00:44:45.877279 6470 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:45.877339 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:45.877588 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:45.879648 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "88f08d97f85d42b29dad35abbe1ab14d"
format_stamp: "Formatted at 2026-09-27 00:44:45 on dist-test-slave-l5sb"
server_key: "e8dcec12cccbabe0840d284a67e35ae3"
server_key_iv: "20f4da1ea4a8676982bda1d654b5f226"
server_key_version: "encryptionkey@0"
I20260927 00:44:45.880195 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "88f08d97f85d42b29dad35abbe1ab14d"
format_stamp: "Formatted at 2026-09-27 00:44:45 on dist-test-slave-l5sb"
server_key: "e8dcec12cccbabe0840d284a67e35ae3"
server_key_iv: "20f4da1ea4a8676982bda1d654b5f226"
server_key_version: "encryptionkey@0"
I20260927 00:44:45.883729 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:44:45.886057 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:45.887229 6470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:45.887364 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "88f08d97f85d42b29dad35abbe1ab14d"
format_stamp: "Formatted at 2026-09-27 00:44:45 on dist-test-slave-l5sb"
server_key: "e8dcec12cccbabe0840d284a67e35ae3"
server_key_iv: "20f4da1ea4a8676982bda1d654b5f226"
server_key_version: "encryptionkey@0"
I20260927 00:44:45.887480 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:45.902185 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:45.905318 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:45.905519 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:45.912226 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33789
I20260927 00:44:45.912501 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33789 every 8 connection(s)
I20260927 00:44:45.913167 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:45.914170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6470
I20260927 00:44:45.914321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:45.914678 392 external_mini_cluster.cc:1428] Setting key c2f6c638e6e181caae2702604dc970c9
I20260927 00:44:45.916455 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:45.922744 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Bootstrap starting.
Sep 27 00:44:45 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:45.924855 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:45.925531 6538 log.cc:824] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:45.927327 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: No bootstrap required, opened a new log
I20260927 00:44:45.928377 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:45.915962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.98.62:33789)
0927 00:44:45.916358 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:45.916381 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:45.916415 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:44:45.917081 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:45.917096 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:45.917397 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:45.917658 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:45.918547 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:45.919529 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:45.920630 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:45.921099 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:45.921206 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:45.923898 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:45.923903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:45.923924 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:45.923949 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:45.925620 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:45.926104 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:45.926110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:45.926122 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:45.926240 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:45.926525 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:45.926528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:45.926529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:45.926895 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:45.927089 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:45.927370 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:44:45.927727 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260927 00:44:45.930996 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } }
I20260927 00:44:45.931227 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:45.931313 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f08d97f85d42b29dad35abbe1ab14d, State: Initialized, Role: FOLLOWER
I20260927 00:44:45.931737 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } }
I20260927 00:44:45.931885 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:45.931965 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:45.932061 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:45.932879 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } }
I20260927 00:44:45.933197 6538 leader_election.cc:302] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f08d97f85d42b29dad35abbe1ab14d; no voters:
I20260927 00:44:45.933473 6538 leader_election.cc:288] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:45.933594 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:45.933840 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 LEADER]: Becoming Leader. State: Replica: 88f08d97f85d42b29dad35abbe1ab14d, State: Running, Role: LEADER
I20260927 00:44:45.934183 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } }
I20260927 00:44:45.934634 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:45.935750 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } }
I20260927 00:44:45.935873 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:45.936230 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f08d97f85d42b29dad35abbe1ab14d. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } }
I20260927 00:44:45.936365 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
W20260927 00:44:45.939182 6558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:45.939266 6558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:45.939324 6552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:45.939992 6552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:45.944589 6552 catalog_manager.cc:1347] Generated new cluster ID: 9446a0a341d04ab4b442b41395102c56
I20260927 00:44:45.944648 6552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:45.953814 6552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:45.954959 6552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:45.962134 6552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Generated new TSK 0
I20260927 00:44:45.962898 6552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:44:45.994616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:46.102373 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:46.102643 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:46.102687 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:46.106738 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:46.106796 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:46.106855 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:46.112150 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:46.113178 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:46.114216 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:46.123803 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.123857 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.124022 6564 server_base.cc:1034] running on GCE node
W20260927 00:44:46.123795 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.124496 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:46.125137 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:46.126314 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790469886126304 us; error 29 us; skew 500 ppm
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:46.129081 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:44:46.130024 6564 webserver.cc:460] Webserver started at http://127.0.98.1:35913/ using document root <none> and password file <none>
I20260927 00:44:46.130535 6564 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.130591 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.130749 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:46.132642 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "17e867ec41ee449cbf454c7595b71a49"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "31f66865a54ec7135697965850de22eb"
server_key_iv: "d2da7f8602ecd019381312e4eb1039e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.133127 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17e867ec41ee449cbf454c7595b71a49"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "31f66865a54ec7135697965850de22eb"
server_key_iv: "d2da7f8602ecd019381312e4eb1039e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.136865 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:44:46.139166 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.140480 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:46.140570 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "17e867ec41ee449cbf454c7595b71a49"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "31f66865a54ec7135697965850de22eb"
server_key_iv: "d2da7f8602ecd019381312e4eb1039e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.140652 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.158507 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:46.161801 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:46.161952 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:46.162467 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:46.163203 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:46.163249 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.163287 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:46.163302 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.171958 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36323
I20260927 00:44:46.171998 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36323 every 8 connection(s)
I20260927 00:44:46.172786 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:46.180572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6564
I20260927 00:44:46.180713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:46.180986 392 external_mini_cluster.cc:1428] Setting key 1bdc424f8f64ed397cbdbc727af408c1
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:46.186421 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.174500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50359 (local address 127.0.98.62:33789)
0927 00:44:46.174645 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.174649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.175796 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 00:44:46.176663 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.176666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.176712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.176784 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.177964 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.178481 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.179579 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.179793 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.179865 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.182502 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.182505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.182507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.182528 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.184298 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.184772 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.184778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.184783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.184866 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.185282 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.185284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.185285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.185445 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.185521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.186168 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.186290 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:44:46.187547 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.174849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:50359)
0927 00:44:46.175655 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.175694 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.176482 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.176945 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.176951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.177285 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.177759 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.177802 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.178609 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.178613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.179478 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.179487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.179909 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.179911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.180048 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.180639 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.180655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.182348 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.184415 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.184419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.184430 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.184674 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.184990 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.184996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.185011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.185189 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.185545 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.185555 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.185895 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.186111 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.186471 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":499,"threads_started":1}
I20260927 00:44:46.188759 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:46.189016 6693 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:46.189576 6693 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:46.191149 6502 ts_manager.cc:194] Registered new tserver with Master: 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:46.192298 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:44:46.208967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:46.318493 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:46.318751 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:46.318794 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:46.323096 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:46.323160 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:46.323220 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:46.328488 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:46.329486 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:46.330627 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:46.340432 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.340478 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.340478 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.340636 6698 server_base.cc:1034] running on GCE node
I20260927 00:44:46.341153 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:46.341892 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:46.343075 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790469886343057 us; error 27 us; skew 500 ppm
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:46.345908 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:44:46.346704 6698 webserver.cc:460] Webserver started at http://127.0.98.2:34447/ using document root <none> and password file <none>
I20260927 00:44:46.347204 6698 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.347259 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.347424 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:46.349354 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "38bcfb9f41f34461a8dedc7c1ef6073a"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "f16a6d7d981850eae2db64b9b1de2011"
server_key_iv: "ff61eab8ae3e037e49ec24a6506ba72f"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.349954 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38bcfb9f41f34461a8dedc7c1ef6073a"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "f16a6d7d981850eae2db64b9b1de2011"
server_key_iv: "ff61eab8ae3e037e49ec24a6506ba72f"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.353461 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:46.355774 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.356956 6698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:46.357046 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "38bcfb9f41f34461a8dedc7c1ef6073a"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "f16a6d7d981850eae2db64b9b1de2011"
server_key_iv: "ff61eab8ae3e037e49ec24a6506ba72f"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.357124 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.375650 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:46.378906 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:46.379060 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:46.379598 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:46.380336 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:46.380381 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.380427 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:46.380450 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.388999 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35233
I20260927 00:44:46.389051 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35233 every 8 connection(s)
I20260927 00:44:46.389962 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:46.394865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6698
I20260927 00:44:46.394963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:46.395249 392 external_mini_cluster.cc:1428] Setting key db404757b2327ac0c8f14e939bf40a3b
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:46.404029 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.391759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57459 (local address 127.0.98.62:33789)
0927 00:44:46.391937 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.391941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.392732 (+ 791us) server_negotiation.cc:407] Connection header received
0927 00:44:46.394120 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.394123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.394168 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.394235 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.395491 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.396109 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.397337 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.397752 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.397899 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.400472 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.400475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.400495 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.400528 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.402036 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.402526 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.402531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.402533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.402612 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.402960 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.402963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.402965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.403139 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.403243 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.403744 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.403861 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:46.404824 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.392167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:57459)
0927 00:44:46.392586 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.392627 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.393879 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.394402 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.394408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.394755 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.395359 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.395373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.396229 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.396232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.397215 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.397223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.397918 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.397920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.398052 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.398652 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.398674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.400300 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.402149 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.402155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.402170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.402425 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.402708 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.402711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.402713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.402846 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.403240 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.403247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.403491 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.403685 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.404003 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260927 00:44:46.406064 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:46.406359 6827 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 00:44:46.406927 6827 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
I20260927 00:44:46.408021 6502 ts_manager.cc:194] Registered new tserver with Master: 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:46.408821 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:44:46.422608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:46.531484 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:46.531723 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:46.531766 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:46.535820 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:46.535869 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:46.535967 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:46.541690 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:46.542688 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:46.543615 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:46.552977 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.552976 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.553179 6832 server_base.cc:1034] running on GCE node
W20260927 00:44:46.553078 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.553673 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:46.554564 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:46.555780 6832 hybrid_clock.cc:648] HybridClock initialized: now 1790469886555749 us; error 42 us; skew 500 ppm
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:46.558579 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:44:46.559482 6832 webserver.cc:460] Webserver started at http://127.0.98.3:46477/ using document root <none> and password file <none>
I20260927 00:44:46.560050 6832 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.560112 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.560343 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:46.562415 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "aad31d43949c4bfea794419be955e623"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "811d5419c975b9890aa2c06498bd04e2"
server_key_iv: "d79c5c98953a42d3844f356f03fd1a49"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.562949 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aad31d43949c4bfea794419be955e623"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "811d5419c975b9890aa2c06498bd04e2"
server_key_iv: "d79c5c98953a42d3844f356f03fd1a49"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.566495 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:44:46.568820 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.570052 6832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 00:44:46.570276 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "aad31d43949c4bfea794419be955e623"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "811d5419c975b9890aa2c06498bd04e2"
server_key_iv: "d79c5c98953a42d3844f356f03fd1a49"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.570410 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.589145 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:46.592010 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:46.592206 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:46.592754 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:46.593528 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:46.593597 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.593667 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:46.593708 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.602363 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42557
I20260927 00:44:46.602386 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42557 every 8 connection(s)
I20260927 00:44:46.603180 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:46.608834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6832
I20260927 00:44:46.608945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:46.609215 392 external_mini_cluster.cc:1428] Setting key ab377e33e35f93a32088ea4eb2972ec8
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:46.616675 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.604835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49809 (local address 127.0.98.62:33789)
0927 00:44:46.604979 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.604983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.605859 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:44:46.607178 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.607181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.607226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.607301 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.608434 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.609202 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.610586 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.610804 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.610918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.613230 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.613236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.613244 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.613265 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.614706 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.615197 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.615202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.615204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.615281 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.615645 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.615649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.615651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.615834 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.615967 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.616413 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.616521 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:46.617386 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.605248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:49809)
0927 00:44:46.605708 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.605757 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.607001 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.607493 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.607499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.607840 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.608307 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.608321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.609379 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.609383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.610464 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.610474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.610955 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.610959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.611116 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.611734 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.611748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.613082 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.614828 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.614833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.614847 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.615097 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.615382 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.615386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.615388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.615545 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.615967 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.615978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.616195 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.616373 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.616667 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":138,"threads_started":1}
I20260927 00:44:46.618386 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:46.618604 6961 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:46.619059 6961 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
I20260927 00:44:46.619992 6502 ts_manager.cc:194] Registered new tserver with Master: aad31d43949c4bfea794419be955e623 (127.0.98.3:42557)
I20260927 00:44:46.620522 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49809
I20260927 00:44:46.622802 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 00:44:46.646965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39033
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33789,127.0.98.61:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:46.754793 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:46.755052 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:46.755093 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:46.759023 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:46.759075 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:46.759095 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:46.759112 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:46.759127 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:46.764329 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33789,127.0.98.61:39033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
W20260927 00:44:46.764501 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33789,127.0.98.61:39033'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:44:46.765308 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:46.766299 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:46.775717 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.775704 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.776199 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.776345 6966 server_base.cc:1034] running on GCE node
I20260927 00:44:46.776959 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:46.778316 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:46.779538 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790469886779526 us; error 23 us; skew 500 ppm
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:46.782260 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:44:46.783041 6966 webserver.cc:460] Webserver started at http://127.0.98.61:46199/ using document root <none> and password file <none>
I20260927 00:44:46.783530 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.783583 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.783735 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:46.785609 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/instance:
uuid: "ddd7188520aa4c03857953e59e405e29"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "a4ee59528d702e25522f2b2eca64ffd6"
server_key_iv: "c097de8d68f778b33374c36ee28bbfbd"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.786170 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal/instance:
uuid: "ddd7188520aa4c03857953e59e405e29"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "a4ee59528d702e25522f2b2eca64ffd6"
server_key_iv: "c097de8d68f778b33374c36ee28bbfbd"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.789623 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:46.792135 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.793294 6966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:46.793440 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "ddd7188520aa4c03857953e59e405e29"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "a4ee59528d702e25522f2b2eca64ffd6"
server_key_iv: "c097de8d68f778b33374c36ee28bbfbd"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.793543 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.807466 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:46.810972 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:46.811172 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:46.817500 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39033
I20260927 00:44:46.817840 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39033 every 8 connection(s)
I20260927 00:44:46.818507 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:44:46.821159 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:46.822567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 6966
I20260927 00:44:46.825145 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:46.836447 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.826811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.62:33789)
0927 00:44:46.826954 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.826958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.827238 (+ 280us) server_negotiation.cc:407] Connection header received
0927 00:44:46.827351 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.827354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.827411 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.827479 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.828233 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.829028 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.830251 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.830589 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.830712 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.833171 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.833179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.833181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.833204 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.834600 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.835060 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.835065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.835067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.835141 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.835500 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.835505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.835507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.835710 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.835817 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.836171 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.836279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:44:46.837213 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.826739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:58970)
0927 00:44:46.827113 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.827157 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.827263 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.827542 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.827547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.827962 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.828116 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.828124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.829176 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.829179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.830123 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.830133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.830747 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.830750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.830826 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.831339 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.831354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.833034 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.834703 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.834713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.834722 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.834960 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.835231 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.835235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.835237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.835406 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.835814 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.835819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.835969 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.836129 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.836415 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":126,"threads_started":1}
I20260927 00:44:46.837217 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.824023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.0.98.61:39033)
0927 00:44:46.824444 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.824457 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.824490 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:44:46.825074 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.825089 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.825387 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.825754 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.826546 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.827995 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.829162 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.829870 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.830007 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.832696 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.832702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.832721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.832748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.834618 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.835088 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.835093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.835095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.835197 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.835487 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.835492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.835493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.835890 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.836110 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.836344 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.836473 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260927 00:44:46.838501 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:44:46.846029 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.838956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39820)
0927 00:44:46.839115 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.839145 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.839245 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.839462 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.839465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.839614 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.839685 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.839689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.840681 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.840687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.841173 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.841179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.841621 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.841625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.841713 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.842197 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.842212 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.843306 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.844840 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.844842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.844844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.845015 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.845233 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.845235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.845236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.845303 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.845593 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.845595 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.845661 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.845736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.845905 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 00:44:46.846100 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.839020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.98.61:39033)
0927 00:44:46.839133 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.839137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.839209 (+ 72us) server_negotiation.cc:407] Connection header received
0927 00:44:46.839329 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.839333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.839366 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.839454 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.839791 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.840579 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.841286 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.841524 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.841625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.843436 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.843438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.843440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.843459 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.844751 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.845094 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.845096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.845097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.845160 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.845373 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.845375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.845376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.845521 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.845589 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.845846 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.845979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":37}
I20260927 00:44:46.848750 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Bootstrap starting.
I20260927 00:44:46.850978 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:46.851658 7034 log.cc:824] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:46.853402 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: No bootstrap required, opened a new log
I20260927 00:44:46.856405 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } }
I20260927 00:44:46.856622 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:46.856709 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd7188520aa4c03857953e59e405e29, State: Initialized, Role: FOLLOWER
I20260927 00:44:46.857106 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } }
I20260927 00:44:46.857527 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } } }
I20260927 00:44:46.857630 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:46.858251 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:44:46.862891 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:46.862977 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:46.864737 6966 master_runner.cc:428] Detected that this master ddd7188520aa4c03857953e59e405e29 is joining an existing cluster
I20260927 00:44:46.864810 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39033
I20260927 00:44:46.875217 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.867000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:58986)
0927 00:44:46.867268 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.867282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.867384 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.867693 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.867696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.867968 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.868098 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.868105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.869124 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.869127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.869868 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.869876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.870331 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.870335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.870470 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.870989 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.871001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.871400 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.873454 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.873458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.873461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.873812 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.874147 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.874149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.874151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.874231 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.874723 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.874731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.874800 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.874908 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.875031 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":65,"thread_start_us":98,"threads_started":1}
I20260927 00:44:46.875201 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.867074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.98.62:33789)
0927 00:44:46.867194 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.867199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.867368 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:44:46.867498 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.867502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.867562 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.867643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.868226 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.869007 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.869977 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.870207 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.870329 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.871576 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.871579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.871582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.871608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.873302 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.873948 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.873952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.873954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.874030 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.874378 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.874381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.874383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.874596 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.874684 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.874959 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.875066 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:44:46.876478 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.867324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.98.61:39033)
0927 00:44:46.867442 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.867447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.867796 (+ 349us) server_negotiation.cc:407] Connection header received
0927 00:44:46.867886 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.867892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.867939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.868019 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.868508 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.869197 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.870389 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.870740 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.870810 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.872020 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.872025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.872028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.872050 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.873828 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.874265 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.874270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.874273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.874355 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.874665 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.874668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.874669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.874855 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.874945 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.875199 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.875337 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260927 00:44:46.876969 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.867000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39828)
0927 00:44:46.867713 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.867726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.867814 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.868050 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.868053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.868251 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.868382 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.868390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.869306 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.869309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.870265 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.870275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.870932 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.870935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.871031 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.871480 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.871494 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.871870 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.873925 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.873930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.873933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.874160 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.874442 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.874445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.874447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.874569 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.874977 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.874980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.875053 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.876695 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.876813 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":454,"thread_start_us":86,"threads_started":1}
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:44:46.888938 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.879761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.98.61:39033)
0927 00:44:46.879871 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.879874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.880573 (+ 699us) server_negotiation.cc:407] Connection header received
0927 00:44:46.880738 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.880740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.880790 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.880853 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.881573 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.882086 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.883281 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.883477 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.883554 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.885814 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.885818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.885821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.885850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.887235 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.887700 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.887705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.887706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.887787 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.888063 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.888067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.888069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.888234 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.888342 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.888659 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.888763 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:46.888965 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.879724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39838)
0927 00:44:46.880453 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.880501 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.880623 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.880908 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.880914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.881335 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.881471 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.881478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.882181 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.882185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.883166 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.883176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.883579 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.883583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.883685 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.884164 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.884178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.885651 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.887362 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.887370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.887376 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.887600 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.887884 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.887887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.887889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.887974 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.888342 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.888351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.888469 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.888612 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.888798 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":74,"threads_started":1}
I20260927 00:44:46.890925 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39033: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ddd7188520aa4c03857953e59e405e29"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39033
}
attrs {
promote: true
}
}
dest_uuid: "88f08d97f85d42b29dad35abbe1ab14d"
cas_config_opid_index: -1
I20260927 00:44:46.891649 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } }
I20260927 00:44:46.894044 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:46.895241 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:44:46.893471 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ddd7188520aa4c03857953e59e405e29 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } }
I20260927 00:44:46.896974 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39033
I20260927 00:44:46.897226 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:46.897508 6966 master.cc:537] Master@127.0.98.61:39033 shutting down...
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:44:46.901316 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } }
I20260927 00:44:46.901530 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
W20260927 00:44:46.901577 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:44:46.902362 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:46.902665 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:46.902819 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 00:44:46.908188 6966 master.cc:552] Master@127.0.98.61:39033 shutdown complete.
I20260927 00:44:46.908234 6966 master_runner.cc:305] Clearing existing system tablet
I20260927 00:44:46.908476 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.908547 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.909598 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:44:46.910523 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.910231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39852)
0927 00:44:46.910362 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.910430 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:44:46.910703 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d -> Peer ddd7188520aa4c03857953e59e405e29 (127.0.98.61:39033): Couldn't send request to peer ddd7188520aa4c03857953e59e405e29. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:44:46.917908 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 00:44:46.919045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36059
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:46.924152 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.924429 6966 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.000s sys 0.001s
I20260927 00:44:46.924518 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "ddd7188520aa4c03857953e59e405e29"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "a4ee59528d702e25522f2b2eca64ffd6"
server_key_iv: "c097de8d68f778b33374c36ee28bbfbd"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.926702 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:46.928647 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:44:46.928753 6966 log.cc:1196] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:44:46.928999 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Deleting consensus metadata
I20260927 00:44:46.929186 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:33789
I20260927 00:44:46.930327 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33789
I20260927 00:44:46.938524 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.930912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.0.98.62:33789)
0927 00:44:46.931031 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:46.931035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:46.931272 (+ 237us) server_negotiation.cc:407] Connection header received
0927 00:44:46.931456 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:46.931459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:46.931510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:46.931593 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:46.932244 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.932761 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.933590 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.933873 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.933979 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.935232 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:46.935236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:46.935239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:46.935264 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:46.936817 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.937297 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.937304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.937306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.937390 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:46.937638 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:46.937642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:46.937644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:46.937880 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:46.937983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:46.938238 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:44:46.938356 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:46.938551 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:46.930861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:58990)
0927 00:44:46.931168 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:46.931180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:46.931318 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:46.931609 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:46.931613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:46.931835 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:46.932120 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.932126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.932868 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.932875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.933469 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:46.933477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:46.933982 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:46.933986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:46.934078 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:46.934673 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:46.934690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:46.935108 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:46.936926 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.936930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.936932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.937184 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.937481 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:46.937483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:46.937484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:46.937548 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:46.937978 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:46.937981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:46.938079 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:44:46.938185 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:44:46.938329 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":89,"threads_started":1}
I20260927 00:44:46.939344 6522 tablet_copy_service.cc:140] P 88f08d97f85d42b29dad35abbe1ab14d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ddd7188520aa4c03857953e59e405e29 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58990)
I20260927 00:44:46.939512 6522 tablet_copy_service.cc:161] P 88f08d97f85d42b29dad35abbe1ab14d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ddd7188520aa4c03857953e59e405e29 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58990: session id = ddd7188520aa4c03857953e59e405e29-00000000000000000000000000000000
I20260927 00:44:46.940923 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:44:46.942358 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:44:46.944224 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: tablet copy: Starting download of 0 data blocks...
I20260927 00:44:46.944429 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: tablet copy: Starting download of 1 WAL segments...
I20260927 00:44:46.946044 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:44:46.947523 6522 tablet_copy_service.cc:342] P 88f08d97f85d42b29dad35abbe1ab14d: Request end of tablet copy session ddd7188520aa4c03857953e59e405e29-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58990
I20260927 00:44:46.947630 6522 tablet_copy_service.cc:433] P 88f08d97f85d42b29dad35abbe1ab14d: ending tablet copy session ddd7188520aa4c03857953e59e405e29-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ddd7188520aa4c03857953e59e405e29
I20260927 00:44:46.954010 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:46.955751 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.955932 6966 server_base.cc:1034] running on GCE node
W20260927 00:44:46.955724 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:46.956213 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:46.956491 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:46.956801 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:46.957938 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790469886957935 us; error 22 us; skew 500 ppm
Sep 27 00:44:46 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:46.959430 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:44:46.959969 6966 webserver.cc:460] Webserver started at http://127.0.98.61:45687/ using document root <none> and password file <none>
I20260927 00:44:46.960189 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:46.960244 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:46.961454 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:46.962392 7086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:46.962616 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:46.962711 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "ddd7188520aa4c03857953e59e405e29"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "a4ee59528d702e25522f2b2eca64ffd6"
server_key_iv: "c097de8d68f778b33374c36ee28bbfbd"
server_key_version: "encryptionkey@0"
I20260927 00:44:46.962815 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:46.972684 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:46.975126 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:46.980494 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39033
I20260927 00:44:46.981114 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:44:46.981585 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39033 every 8 connection(s)
I20260927 00:44:46.982833 7139 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:44:46.983670 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Bootstrap starting.
I20260927 00:44:46.992681 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:46.993009 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Bootstrap complete.
I20260927 00:44:46.993525 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } }
I20260927 00:44:46.993664 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ddd7188520aa4c03857953e59e405e29, State: Initialized, Role: LEARNER
I20260927 00:44:46.993827 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } }
I20260927 00:44:46.994199 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } }
I20260927 00:44:46.994325 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:46.994393 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:44:46.997515 7153 catalog_manager.cc:1259] Loaded cluster ID: 9446a0a341d04ab4b442b41395102c56
I20260927 00:44:46.997627 7153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: loading cluster ID for follower catalog manager: success
I20260927 00:44:46.999328 7153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: acquiring CA information for follower catalog manager: success
I20260927 00:44:47.000458 7153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:44:47.067943 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:47.068223 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:47.068293 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:47.072474 7069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:47.072561 7069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:47.072598 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:47.072656 7069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:47.072693 7069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:47.078155 7069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:47.079232 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:47.080260 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:47.088622 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.088622 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.088667 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.088851 7069 server_base.cc:1034] running on GCE node
I20260927 00:44:47.089506 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:47.090664 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:47.091845 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790469887091817 us; error 44 us; skew 500 ppm
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:47.094717 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:44:47.095777 7069 webserver.cc:460] Webserver started at http://127.0.98.60:35351/ using document root <none> and password file <none>
I20260927 00:44:47.096436 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.096524 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.096735 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:47.098812 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/instance:
uuid: "e1e622b6712d43d6a7e464e2fc21309d"
format_stamp: "Formatted at 2026-09-27 00:44:47 on dist-test-slave-l5sb"
server_key: "45e7a1a60d152cdfb9e0f08a54c8e3a5"
server_key_iv: "095290f66469b86c0d4e72e5f5ab227b"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.099400 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal/instance:
uuid: "e1e622b6712d43d6a7e464e2fc21309d"
format_stamp: "Formatted at 2026-09-27 00:44:47 on dist-test-slave-l5sb"
server_key: "45e7a1a60d152cdfb9e0f08a54c8e3a5"
server_key_iv: "095290f66469b86c0d4e72e5f5ab227b"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.102967 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:47.105273 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.106545 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:47.106681 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "e1e622b6712d43d6a7e464e2fc21309d"
format_stamp: "Formatted at 2026-09-27 00:44:47 on dist-test-slave-l5sb"
server_key: "45e7a1a60d152cdfb9e0f08a54c8e3a5"
server_key_iv: "095290f66469b86c0d4e72e5f5ab227b"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.106786 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.135350 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:47.138770 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:47.138988 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:47.146668 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36059
I20260927 00:44:47.146944 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36059 every 8 connection(s)
I20260927 00:44:47.147753 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:44:47.148953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7069
I20260927 00:44:47.151437 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:47.156055 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:47.165902 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.150430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.0.98.60:36059)
0927 00:44:47.150844 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.150863 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.150908 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:44:47.151759 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.151783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.152186 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.152549 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.153206 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.154723 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.155613 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.156266 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.156436 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.159336 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.159342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.159355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.159388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.162560 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.163061 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.163069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.163077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.163194 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.163496 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.163498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.163499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.163928 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.164184 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.164470 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.164764 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260927 00:44:47.167375 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.157856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59004)
0927 00:44:47.158246 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.158287 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.158618 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.158969 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.158975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.159390 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.159552 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.159560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.160592 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.160595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.161400 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.161410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.161885 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.161889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.162002 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.162441 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.162459 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.163980 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.165924 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.165934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.165940 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.166223 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.166478 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.166481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.166483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.166570 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.166879 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.166884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.167004 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.167114 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.167236 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260927 00:44:47.167448 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.157989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.0.98.62:33789)
0927 00:44:47.158133 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.158137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.158363 (+ 226us) server_negotiation.cc:407] Connection header received
0927 00:44:47.158731 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.158735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.158789 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.158876 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.159703 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.160490 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.161515 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.161720 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.161883 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.164199 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.164205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.164208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.164233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.165812 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.166316 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.166319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.166320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.166391 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.166648 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.166649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.166650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.166792 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.166859 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.167198 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.167310 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:44:47.169245 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:44:47.177295 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.169674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39854)
0927 00:44:47.169839 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.169853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.169920 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.170244 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.170246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.170374 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.170465 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.170470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.171296 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.171299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.171885 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.171895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.172296 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.172299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.172382 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.172779 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.172793 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.174135 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.175901 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.175905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.175910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.176100 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.176374 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.176376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.176378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.176469 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.176826 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.176829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.176914 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.177020 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.177142 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:44:47.177346 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.169814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.98.61:39033)
0927 00:44:47.170061 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.170065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.170076 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:47.170109 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.170111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.170148 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.170218 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.170559 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.171199 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.171985 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.172198 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.172273 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.174284 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.174287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.174289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.174310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.175777 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.176195 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.176200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.176202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.176279 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.176563 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.176566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.176567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.176734 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.176837 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.177079 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.177193 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260927 00:44:47.178087 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:44:47.185194 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.178496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.0.1:48886)
0927 00:44:47.178600 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.178614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.178688 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.178858 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.178860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.178981 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.179043 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.179047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.180014 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.180016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.180488 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.180495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.180894 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.180897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.180961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.181323 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.181334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.182478 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.183999 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.184001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.184002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.184171 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.184385 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.184387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.184389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.184453 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.184767 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.184770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.184843 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.184937 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.185038 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:44:47.185205 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.178571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.98.60:36059)
0927 00:44:47.178671 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.178675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.178694 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:44:47.178742 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.178743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.178773 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.178831 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.179128 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.179927 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.180573 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.180776 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.180846 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.182587 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.182590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.182592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.182610 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.183906 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.184248 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.184251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.184252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.184314 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.184522 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.184524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.184525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.184660 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.184732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.184982 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.185088 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":40}
I20260927 00:44:47.187971 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Bootstrap starting.
I20260927 00:44:47.190218 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:47.190918 7221 log.cc:824] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:47.192698 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: No bootstrap required, opened a new log
I20260927 00:44:47.194677 6693 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
I20260927 00:44:47.196116 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } }
I20260927 00:44:47.196349 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:47.196415 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e622b6712d43d6a7e464e2fc21309d, State: Initialized, Role: FOLLOWER
I20260927 00:44:47.196765 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } }
I20260927 00:44:47.197170 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } } }
I20260927 00:44:47.197305 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:47.197889 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:44:47.202332 7238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:44:47.202409 7238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:44:47.205282 7069 master_runner.cc:428] Detected that this master e1e622b6712d43d6a7e464e2fc21309d is joining an existing cluster
I20260927 00:44:47.205356 7069 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36059
I20260927 00:44:47.219228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6564
I20260927 00:44:47.225724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36323
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33789
--webserver_port=35913
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:47.236140 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.207465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.0.98.61:39033)
0927 00:44:47.207583 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.207589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.208589 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 00:44:47.208690 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.208693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.208732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.208823 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.209241 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.210082 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.211243 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.211929 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.212012 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.214183 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.214188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.214191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.214225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.216042 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.234501 (+ 18459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.234506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.234508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.234614 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.234990 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.234993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.234995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.235255 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.235331 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.235787 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.235955 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:47.236177 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.207692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.98.62:33789)
0927 00:44:47.207893 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.207898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.207912 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:47.207951 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.207954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.207999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.208087 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.208496 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.209330 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.210567 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.210902 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.210970 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.212484 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.212489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.212493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.212519 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.214276 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.214837 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.214844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.214848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.214933 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.234600 (+ 19667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.234607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.234611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.234927 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.235027 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.235888 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.236017 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 00:44:47.236727 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.207391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39856)
0927 00:44:47.208508 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.208520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.208618 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.208846 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.208848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.209049 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.209133 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.209138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.210203 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.210206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.211129 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.211135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.212156 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.212159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.212279 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.212998 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.213019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.213739 (+ 720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.216160 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.216163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.216165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.234304 (+ 18139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.234738 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.234741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.234743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.234885 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.235417 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.235420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.235574 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.236383 (+ 809us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.236538 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":29,"spinlock_wait_cycles":3328,"thread_start_us":100,"threads_started":1}
I20260927 00:44:47.237246 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.207282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59018)
0927 00:44:47.207734 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.207750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.207834 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.208092 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.208094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.208253 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.208389 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.208398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.209443 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.209447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.210445 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.210454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.211125 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.211128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.211266 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.211839 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.211854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.212311 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.214405 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.214410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.214415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.214719 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.215056 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.215062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.215067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.234009 (+ 18942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.235113 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.235119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.235242 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.236738 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.237071 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260927 00:44:47.238468 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.207753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.0.98.60:36059)
0927 00:44:47.207841 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.207845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.210050 (+ 2205us) server_negotiation.cc:407] Connection header received
0927 00:44:47.210188 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.210192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.210227 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.210270 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.210808 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.211547 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.212717 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.213039 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.213104 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.214734 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.214737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.214740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.214759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.216284 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.235458 (+ 19174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.235461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.235463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.235550 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.237544 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.237548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.237550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.237744 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.237847 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.238220 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.238321 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":2816}
I20260927 00:44:47.239107 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.208235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.0.1:48902)
0927 00:44:47.209977 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.209990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.210098 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.210392 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.210395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.210587 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.210686 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.210693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.211745 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.211748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.212598 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.212607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.213224 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.213228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.213310 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.214194 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.214208 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.214610 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.216377 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.216379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.216380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.234450 (+ 18070us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.235638 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.235641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.235642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.235709 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.237888 (+ 2179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.237890 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.237960 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.238813 (+ 853us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.238946 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1673,"thread_start_us":88,"threads_started":1}
I20260927 00:44:47.244285 7227 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:47.244439 7227 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } }
I20260927 00:44:47.245450 7227 leader_election.cc:288] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), ddd7188520aa4c03857953e59e405e29 (127.0.98.61:39033)
I20260927 00:44:47.246423 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1e622b6712d43d6a7e464e2fc21309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd7188520aa4c03857953e59e405e29" is_pre_election: true
I20260927 00:44:47.246483 6512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1e622b6712d43d6a7e464e2fc21309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f08d97f85d42b29dad35abbe1ab14d" is_pre_election: true
I20260927 00:44:47.246606 7113 raft_consensus.cc:1799] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Handling vote request from an unknown peer e1e622b6712d43d6a7e464e2fc21309d
I20260927 00:44:47.246728 7113 raft_consensus.cc:2396] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate e1e622b6712d43d6a7e464e2fc21309d for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:44:47.247172 7170 leader_election.cc:302] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1e622b6712d43d6a7e464e2fc21309d; no voters: 88f08d97f85d42b29dad35abbe1ab14d, ddd7188520aa4c03857953e59e405e29
I20260927 00:44:47.248225 7227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:47.249838 7227 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:44:47.250942 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.241886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.0.1:48908)
0927 00:44:47.241989 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.242002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.242092 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.242347 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.242350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.242537 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.242642 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.242649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.243672 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.243676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.244574 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.244584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.245253 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.245257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.245354 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.245851 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.245866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.247370 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.249304 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.249307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.249309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.249549 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.249876 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.249879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.249881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.249965 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.250407 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.250411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.250513 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.250630 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.250790 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 00:44:47.250948 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.241879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.0.98.60:36059)
0927 00:44:47.241983 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.241987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.242065 (+ 78us) server_negotiation.cc:407] Connection header received
0927 00:44:47.242181 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.242184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.242232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.242312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.242762 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.243562 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.244709 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.245046 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.245139 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.247513 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.247517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.247520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.247542 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.249165 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.249668 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.249671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.249673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.249750 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.250079 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.250083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.250085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.250287 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.250378 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.250695 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.250806 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:47.252276 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36059: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36059
}
attrs {
promote: true
}
}
dest_uuid: "88f08d97f85d42b29dad35abbe1ab14d"
cas_config_opid_index: 5
I20260927 00:44:47.252816 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } }
I20260927 00:44:47.253800 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.254522 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:44:47.255053 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:47.255411 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36059
I20260927 00:44:47.255686 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.255818 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:47.256223 7069 master.cc:537] Master@127.0.98.60:36059 shutting down...
W20260927 00:44:47.258683 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d -> Peer e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059): Couldn't send request to peer e1e622b6712d43d6a7e464e2fc21309d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:44:47.259366 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.254272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.0.98.61:39033)
0927 00:44:47.254392 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.254397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.254412 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:47.254454 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.254456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.254511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.254626 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:47.254937 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.256005 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.257885 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.258513 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.258591 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.258732 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.259150 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.259248 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:44:47.260049 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.254042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39866)
0927 00:44:47.254264 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.254278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.254349 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.254691 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.254694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.254712 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:47.254801 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.254810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.256222 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.256225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.257705 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.257715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.258894 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.258898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.258995 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.259035 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.259862 (+ 827us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.259940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 00:44:47.260975 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:44:47.261427 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:44:47.263276 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.264937 7069 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:47.265077 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f08d97f85d42b29dad35abbe1ab14d. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.265198 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:44:47.265182 7069 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:47.265282 7069 tablet_replica.cc:330] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: stopping tablet replica
I20260927 00:44:47.265291 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: true } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.265375 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:44:47.266475 7248 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: attempting to promote NON_VOTER ddd7188520aa4c03857953e59e405e29 to VOTER
I20260927 00:44:47.266929 7248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } }
I20260927 00:44:47.268146 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 LEARNER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:44:47.268599 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 00:44:47.268718 6487 connection.cc:535] client connection to 127.0.98.60:36059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 00:44:47.268776 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d -> Peer e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059): Couldn't send request to peer e1e622b6712d43d6a7e464e2fc21309d. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:44:47.270385 7069 master.cc:552] Master@127.0.98.60:36059 shutdown complete.
I20260927 00:44:47.270434 7069 master_runner.cc:305] Clearing existing system tablet
I20260927 00:44:47.270334 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddd7188520aa4c03857953e59e405e29 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.270682 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.270735 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.270766 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddd7188520aa4c03857953e59e405e29 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.271843 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.271942 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:47.272065 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.272130 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:47.272197 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.272331 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:47.272831 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 00:44:47.273813 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.274053 7069 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.274178 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:47.274276 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "e1e622b6712d43d6a7e464e2fc21309d"
format_stamp: "Formatted at 2026-09-27 00:44:47 on dist-test-slave-l5sb"
server_key: "45e7a1a60d152cdfb9e0f08a54c8e3a5"
server_key_iv: "095290f66469b86c0d4e72e5f5ab227b"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.276193 7069 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:47.278481 7069 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:44:47.278568 7069 log.cc:1196] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:44:47.278813 7069 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Deleting consensus metadata
I20260927 00:44:47.278981 7069 master_runner.cc:315] Copying system tablet from 127.0.98.62:33789
I20260927 00:44:47.280171 7069 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33789
I20260927 00:44:47.289040 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.280742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59020)
0927 00:44:47.280996 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.281011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.281115 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.281391 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.281394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.281605 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.281732 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.281739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.282811 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.282816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.283725 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.283734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.284407 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.284411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.284504 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.285032 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.285046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.285473 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.287474 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.287477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.287479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.287719 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.288002 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.288005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.288007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.288097 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.288497 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.288501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.288606 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.288715 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.288850 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260927 00:44:47.289040 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.281062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.98.62:33789)
0927 00:44:47.281166 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.281171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.281184 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:47.281225 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.281228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.281275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.281362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.281856 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.282681 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.283873 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.284278 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.284420 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.285611 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.285616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.285618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.285642 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.287356 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.287819 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.287823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.287825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.287908 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.288188 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.288191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.288193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.288399 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.288490 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.288779 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.288886 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:44:47.291103 6522 tablet_copy_service.cc:140] P 88f08d97f85d42b29dad35abbe1ab14d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e1e622b6712d43d6a7e464e2fc21309d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59020)
I20260927 00:44:47.291205 6522 tablet_copy_service.cc:161] P 88f08d97f85d42b29dad35abbe1ab14d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e1e622b6712d43d6a7e464e2fc21309d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59020: session id = e1e622b6712d43d6a7e464e2fc21309d-00000000000000000000000000000000
I20260927 00:44:47.292337 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:44:47.293639 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:44:47.295925 7069 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: tablet copy: Starting download of 0 data blocks...
I20260927 00:44:47.296095 7069 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: tablet copy: Starting download of 1 WAL segments...
I20260927 00:44:47.297552 7069 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:44:47.299232 6522 tablet_copy_service.cc:342] P 88f08d97f85d42b29dad35abbe1ab14d: Request end of tablet copy session e1e622b6712d43d6a7e464e2fc21309d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59020
I20260927 00:44:47.299314 6522 tablet_copy_service.cc:433] P 88f08d97f85d42b29dad35abbe1ab14d: ending tablet copy session e1e622b6712d43d6a7e464e2fc21309d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e1e622b6712d43d6a7e464e2fc21309d
I20260927 00:44:47.306051 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:47.307829 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.307917 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.308214 7069 server_base.cc:1034] running on GCE node
W20260927 00:44:47.308288 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.308560 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:47.308847 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:47.309985 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790469887309972 us; error 29 us; skew 500 ppm
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:47.311568 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:44:47.312028 7069 webserver.cc:460] Webserver started at http://127.0.98.60:39241/ using document root <none> and password file <none>
I20260927 00:44:47.312207 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.312247 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.314127 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:47.315075 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.315295 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:47.315364 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "e1e622b6712d43d6a7e464e2fc21309d"
format_stamp: "Formatted at 2026-09-27 00:44:47 on dist-test-slave-l5sb"
server_key: "45e7a1a60d152cdfb9e0f08a54c8e3a5"
server_key_iv: "095290f66469b86c0d4e72e5f5ab227b"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.315460 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.333835 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:47.336165 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:47.341188 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36059
I20260927 00:44:47.341221 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36059 every 8 connection(s)
I20260927 00:44:47.341794 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:44:47.343513 7322 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:44:47.344547 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Bootstrap starting.
I20260927 00:44:47.356021 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:47.356336 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: Bootstrap complete.
I20260927 00:44:47.356928 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } }
I20260927 00:44:47.357076 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1e622b6712d43d6a7e464e2fc21309d, State: Initialized, Role: LEARNER
I20260927 00:44:47.357192 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } }
I20260927 00:44:47.357559 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.357684 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:47.357760 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: LEARNER
I20260927 00:44:47.362005 7336 catalog_manager.cc:1259] Loaded cluster ID: 9446a0a341d04ab4b442b41395102c56
I20260927 00:44:47.362069 7336 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: loading cluster ID for follower catalog manager: success
I20260927 00:44:47.364224 7336 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: acquiring CA information for follower catalog manager: success
I20260927 00:44:47.365569 7336 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:44:47.371802 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:47.372025 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:47.372069 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:47.376318 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:47.376368 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:47.376459 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:47.381623 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35913
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:47.382696 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:47.383716 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:47.393242 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.393307 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.393286 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.393615 7247 server_base.cc:1034] running on GCE node
I20260927 00:44:47.394008 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:47.394588 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:47.395748 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790469887395716 us; error 51 us; skew 500 ppm
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:47.398401 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:44:47.399220 7247 webserver.cc:460] Webserver started at http://127.0.98.1:35913/ using document root <none> and password file <none>
I20260927 00:44:47.399784 7247 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.399889 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.403594 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:44:47.405570 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.406949 7247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:47.407083 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "17e867ec41ee449cbf454c7595b71a49"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "31f66865a54ec7135697965850de22eb"
server_key_iv: "d2da7f8602ecd019381312e4eb1039e3"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.407618 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.411389 6827 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
I20260927 00:44:47.426463 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:47.429847 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:47.430045 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:47.430656 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:47.431489 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:47.431558 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.431629 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:47.431672 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.441720 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36323
I20260927 00:44:47.441751 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36323 every 8 connection(s)
I20260927 00:44:47.442749 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:47.445320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7247
I20260927 00:44:47.445523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6698
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:44:47.454212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35233
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33789
--webserver_port=34447
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:44:47.459751 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.444507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45233 (local address 127.0.98.60:36059)
0927 00:44:47.444755 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.444759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.446786 (+ 2027us) server_negotiation.cc:407] Connection header received
0927 00:44:47.447466 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.447473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.447539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.447621 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.448687 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.449887 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.451266 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.451711 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.451841 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.454664 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.454671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.454675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.454699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.456789 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.457370 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.457373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.457375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.457472 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.458020 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.458024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.458026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.458277 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.458375 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.459398 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.459538 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260927 00:44:47.460727 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.444933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:52115)
0927 00:44:47.446334 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.446398 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.447179 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.447665 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.447671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.448020 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.448532 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.448545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.449358 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.449360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.450629 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.450639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.451353 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.451360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.451568 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.452206 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.452227 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.454245 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.456768 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.456774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.456789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.457378 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.457686 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.457692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.457697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.457904 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.458423 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.458433 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.458845 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.459072 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.459518 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":76,"spinlock_wait_cycles":52480,"thread_start_us":607,"threads_started":1}
I20260927 00:44:47.461186 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.445973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.98.1:49659)
0927 00:44:47.450454 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.450468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.450602 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.450953 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.450956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.451149 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.451259 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.451269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.452833 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.452839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.453916 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.453931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.454920 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.454926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.455049 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.455472 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.455492 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.457160 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.459192 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.459195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.459197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.459434 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.459718 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.459721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.459722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.459808 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.460648 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.460651 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.460741 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.460854 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.461004 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4420,"thread_start_us":85,"threads_started":1}
I20260927 00:44:47.461367 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.446334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49659 (local address 127.0.98.61:39033)
0927 00:44:47.446470 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.446476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.450542 (+ 4066us) server_negotiation.cc:407] Connection header received
0927 00:44:47.450712 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.450717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.450771 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.450821 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.451389 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.452690 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.454057 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.454391 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.454474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.457306 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.457311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.457318 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.457346 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.459071 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.459544 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.459547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.459549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.459624 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.460373 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.460376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.460377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.460542 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.460659 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.461124 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.461235 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:47.461745 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.444506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52115 (local address 127.0.98.62:33789)
0927 00:44:47.444658 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.444663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.446485 (+ 1822us) server_negotiation.cc:407] Connection header received
0927 00:44:47.447379 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.447382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.447438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.447510 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.448666 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.449201 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.450749 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.451091 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.451165 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.454455 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.454461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.454467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.454499 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.456632 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.457488 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.457494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.457499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.457586 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.458014 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.458020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.458025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.458273 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.458369 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.461369 (+ 3000us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.461519 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:47.462509 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.444933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.98.1:45233)
0927 00:44:47.446408 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.446425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.447207 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.447779 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.447785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.448044 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.448536 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.448549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.450018 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.450024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.451124 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.451137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.451977 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.451983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.452112 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.452580 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.452600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.454500 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.456899 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.456904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.456906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.457259 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.457639 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.457644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.457647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.457898 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.458438 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.458442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.458886 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.459069 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.462108 (+ 3039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":2756,"spinlock_wait_cycles":14848,"thread_start_us":116,"threads_started":1}
I20260927 00:44:47.462765 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39033
I20260927 00:44:47.463055 7467 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.463644 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36059
I20260927 00:44:47.463675 7467 heartbeater.cc:511] Master 127.0.98.61:39033 requested a full tablet report, sending...
I20260927 00:44:47.463747 7466 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.463924 7466 heartbeater.cc:511] Master 127.0.98.60:36059 requested a full tablet report, sending...
I20260927 00:44:47.464250 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:47.464342 7465 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.464540 7465 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
I20260927 00:44:47.465230 7279 ts_manager.cc:194] Registered new tserver with Master: 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:47.465230 7103 ts_manager.cc:194] Registered new tserver with Master: 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:47.466204 6502 ts_manager.cc:194] Re-registered known tserver with Master: 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:47.466781 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52115
W20260927 00:44:47.568351 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:47.568612 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:47.568687 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:47.573050 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:47.573118 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:47.573207 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:47.578300 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34447
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:47.579270 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:47.580197 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:47.589661 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.589695 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.589805 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.590107 7473 server_base.cc:1034] running on GCE node
I20260927 00:44:47.590505 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:47.591218 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:47.592398 7473 hybrid_clock.cc:648] HybridClock initialized: now 1790469887592373 us; error 35 us; skew 500 ppm
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:47.595161 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:44:47.596047 7473 webserver.cc:460] Webserver started at http://127.0.98.2:34447/ using document root <none> and password file <none>
I20260927 00:44:47.596580 7473 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.596647 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.600175 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:47.602061 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.603505 7473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:44:47.603617 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "38bcfb9f41f34461a8dedc7c1ef6073a"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "f16a6d7d981850eae2db64b9b1de2011"
server_key_iv: "ff61eab8ae3e037e49ec24a6506ba72f"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.604210 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.615561 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:47.618372 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:47.618522 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:47.619002 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:47.619779 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:47.619824 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.619863 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:47.619884 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.622567 6961 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
I20260927 00:44:47.628888 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35233
I20260927 00:44:47.628906 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35233 every 8 connection(s)
I20260927 00:44:47.629860 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:47.631265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7473
I20260927 00:44:47.631482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6832
I20260927 00:44:47.639784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42557
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33789
--webserver_port=46477
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:44:47.648551 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.631610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36801 (local address 127.0.98.61:39033)
0927 00:44:47.632023 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.632029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.633438 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 00:44:47.634410 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.634416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.634480 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.634565 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.636534 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637431 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639503 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.639841 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.644128 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.644132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.644135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.644164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.646019 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.646714 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.646719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.646722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.646802 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647215 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.647219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.647222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.647473 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.647558 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.648218 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.648345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260927 00:44:47.648746 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.634169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50279 (local address 127.0.98.60:36059)
0927 00:44:47.634467 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.634474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.634488 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:47.634528 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.634531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.634573 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.634642 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.636549 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637342 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639467 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.639807 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639872 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.644560 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.644567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.644570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.644605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.646630 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647169 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.647176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.647179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.647272 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647622 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.647627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.647630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.647882 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.647977 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.648362 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.648526 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260927 00:44:47.648842 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.634002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40255 (local address 127.0.98.62:33789)
0927 00:44:47.634304 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.634311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.634327 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:44:47.634371 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.634377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.634438 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.634514 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.636533 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637342 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639468 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.639798 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.639872 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.645119 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.645123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.645125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.645150 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.647127 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647563 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.647568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.647573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.647653 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647963 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.647966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.647968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.648180 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.648255 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.648597 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.648707 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260927 00:44:47.650162 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.633354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.98.2:50279)
0927 00:44:47.634001 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.634013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.634099 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.635018 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.635022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.635571 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.636373 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.636387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637531 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.637534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.639349 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.639354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.640066 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.640070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.640373 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.641507 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.641519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.644398 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.646767 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.646771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.646775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.647043 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.647406 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.647414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.647506 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.648044 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.648048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.648155 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.649625 (+ 1470us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.649819 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":627,"spinlock_wait_cycles":38656,"thread_start_us":80,"threads_started":1}
I20260927 00:44:47.650151 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.632038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:40255)
0927 00:44:47.632826 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.632874 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.633979 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.634830 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.634839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.635559 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.636372 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.636390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637531 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.637534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.639327 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.639341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.640066 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.640070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.640386 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.641328 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.641344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.644999 (+ 3655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.647237 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.647242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.647449 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.647753 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.647756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.647757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.647833 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.648310 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.648315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.648409 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.649097 (+ 688us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.649204 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":543,"spinlock_wait_cycles":20992,"thread_start_us":667,"threads_started":1}
I20260927 00:44:47.650151 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.632038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.98.2:36801)
0927 00:44:47.633345 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.633361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.634272 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.634830 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.634839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.635589 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.636364 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.636387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.637576 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.637582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.639355 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.639368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.640110 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.640116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.640394 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.641158 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.641190 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.643917 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.646151 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.646161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.646179 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.646591 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.646919 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.646923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.646926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.647102 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.647621 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.647628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.647947 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.648714 (+ 767us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.649069 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"mutex_wait_us":775,"spinlock_wait_cycles":12160,"thread_start_us":131,"threads_started":1}
I20260927 00:44:47.651497 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:47.651571 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36059
I20260927 00:44:47.651727 7603 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.651727 7602 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.651857 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39033
I20260927 00:44:47.651953 7604 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.652324 7604 heartbeater.cc:511] Master 127.0.98.61:39033 requested a full tablet report, sending...
I20260927 00:44:47.652324 7603 heartbeater.cc:511] Master 127.0.98.60:36059 requested a full tablet report, sending...
I20260927 00:44:47.652346 7602 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
I20260927 00:44:47.653446 6502 ts_manager.cc:194] Re-registered known tserver with Master: 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:47.653576 7279 ts_manager.cc:194] Registered new tserver with Master: 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:47.653584 7103 ts_manager.cc:194] Registered new tserver with Master: 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:47.654196 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40255
W20260927 00:44:47.754175 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:47.754441 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:47.754484 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:47.758677 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:47.758733 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:47.758790 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:47.764137 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46477
--tserver_master_addrs=127.0.98.62:33789,127.0.98.61:39033,127.0.98.60:36059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:47.765084 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:47.766160 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:47.775947 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.775970 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:47.776118 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:47.776346 7609 server_base.cc:1034] running on GCE node
I20260927 00:44:47.776711 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:47.777320 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:47.778509 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790469887778494 us; error 24 us; skew 500 ppm
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:47.781157 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:44:47.782055 7609 webserver.cc:460] Webserver started at http://127.0.98.3:46477/ using document root <none> and password file <none>
I20260927 00:44:47.782577 7609 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:47.782642 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:47.786334 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:47.788167 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.789404 7609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:47.789489 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "aad31d43949c4bfea794419be955e623"
format_stamp: "Formatted at 2026-09-27 00:44:46 on dist-test-slave-l5sb"
server_key: "811d5419c975b9890aa2c06498bd04e2"
server_key_iv: "d79c5c98953a42d3844f356f03fd1a49"
server_key_version: "encryptionkey@0"
I20260927 00:44:47.790027 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:47.801726 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:47.804407 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:47.804555 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:47.805020 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:47.805747 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:47.805853 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.805895 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:47.805917 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:47.815662 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42557
I20260927 00:44:47.815730 7738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42557 every 8 connection(s)
I20260927 00:44:47.816718 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:47.826280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7609
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:44:47 dist-test-slave-l5sb krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:44:47.833715 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.820990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57881 (local address 127.0.98.61:39033)
0927 00:44:47.821108 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.821112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.821345 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:44:47.821496 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.821502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.821557 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.821615 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.822353 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.823147 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824469 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824698 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824756 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.828224 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.828228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.828231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.828255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.831562 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.831980 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.831985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.831987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.832064 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.832339 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.832345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.832350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.832564 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.832647 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.833475 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.833569 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:47.834478 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.820730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.98.3:57881)
0927 00:44:47.821268 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.821280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.821378 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.821798 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.821801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.821960 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.822215 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.822222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.823289 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.823292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.824329 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.824340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824936 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.825044 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.825651 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.825668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.827856 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.831668 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.831671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.831673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.831884 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.832164 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.832167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.832169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.832242 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.832692 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.832698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.832981 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.833163 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.833867 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":560,"thread_start_us":76,"threads_started":1}
I20260927 00:44:47.835124 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.820343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.98.3:52803)
0927 00:44:47.820617 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.820632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.820728 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.821200 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.821208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.821616 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.822214 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.822222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.823289 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.823292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.824335 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.824340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824983 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.825088 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.825560 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.825573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.827539 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.830613 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.830619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.830633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.832055 (+ 1422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.832635 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.832638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.832640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.832722 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.833150 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.833153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.833230 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.833316 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.833475 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":93,"spinlock_wait_cycles":13696,"thread_start_us":120,"threads_started":1}
I20260927 00:44:47.835668 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.820453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52803 (local address 127.0.98.60:36059)
0927 00:44:47.820570 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.820574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.820791 (+ 217us) server_negotiation.cc:407] Connection header received
0927 00:44:47.820876 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.820880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.820934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.821016 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.822354 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.823147 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824494 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824847 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824970 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.827752 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.827757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.827760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.827789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.830479 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.832454 (+ 1975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.832457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.832459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.832531 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.832820 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.832823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.832825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.833019 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.833107 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.835333 (+ 2226us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.835466 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:47.836484 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.818533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44653 (local address 127.0.98.62:33789)
0927 00:44:47.818686 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.818691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.819952 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 00:44:47.820923 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.820926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.820979 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.821032 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.822336 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.822862 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.823990 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824210 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824317 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.828303 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.828306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.828308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.828329 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.830629 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.831172 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.831175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.831179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.831257 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.831670 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.831674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.831675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.832323 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.832410 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.836165 (+ 3755us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.836296 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:44:47.836586 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39033
I20260927 00:44:47.836835 7741 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.837011 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.818973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:44653)
0927 00:44:47.819798 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.819837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.820671 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.821198 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.821205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.821595 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:47.822202 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.822216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.822996 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.822999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.823858 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.823866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.824345 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.824348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.824552 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.825300 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:47.825314 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:47.828164 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:47.830737 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.830739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.830741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.831051 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.831374 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:47.831392 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:47.831394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:47.831560 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:47.835822 (+ 4262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:47.835825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:47.835937 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.836619 (+ 682us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.836749 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":74,"spinlock_wait_cycles":6528,"thread_start_us":529,"threads_started":1}
I20260927 00:44:47.837414 7741 heartbeater.cc:511] Master 127.0.98.61:39033 requested a full tablet report, sending...
I20260927 00:44:47.838354 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33789
I20260927 00:44:47.838444 7739 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.838598 7739 heartbeater.cc:511] Master 127.0.98.62:33789 requested a full tablet report, sending...
I20260927 00:44:47.838755 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36059
I20260927 00:44:47.838815 7740 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:47.838838 7103 ts_manager.cc:194] Registered new tserver with Master: aad31d43949c4bfea794419be955e623 (127.0.98.3:42557)
I20260927 00:44:47.838966 7740 heartbeater.cc:511] Master 127.0.98.60:36059 requested a full tablet report, sending...
I20260927 00:44:47.839231 6502 ts_manager.cc:194] Re-registered known tserver with Master: aad31d43949c4bfea794419be955e623 (127.0.98.3:42557)
I20260927 00:44:47.839600 7279 ts_manager.cc:194] Registered new tserver with Master: aad31d43949c4bfea794419be955e623 (127.0.98.3:42557)
I20260927 00:44:47.839908 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44653
I20260927 00:44:47.840276 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.828575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.98.61:39033)
0927 00:44:47.829603 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.830000 (+ 397us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.830015 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:47.830051 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.830054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.830102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.830190 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.831045 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.831772 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.832825 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.833158 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.833245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.835016 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.835019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.835022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.835050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.837057 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.837468 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.837472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.837474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.837555 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.838843 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.838848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.838853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.839065 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.839216 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.839940 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.840089 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":81,"threads_started":1}
I20260927 00:44:47.842274 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.828504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.0.98.60:36059)
0927 00:44:47.829120 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.829131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.829477 (+ 346us) server_negotiation.cc:407] Connection header received
0927 00:44:47.829521 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.829523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.829567 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.829645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.830709 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.831508 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.832663 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.832993 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.833085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.836223 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.836227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.836229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.836253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.838072 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.840655 (+ 2583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.840659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.840660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.840745 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.841056 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.841060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.841061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.841275 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.841400 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.842003 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.842116 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":64,"threads_started":1}
I20260927 00:44:47.845566 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.829005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.98.62:33789)
0927 00:44:47.833747 (+ 4742us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.834092 (+ 345us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.834107 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:47.834144 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.834147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.834199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.839272 (+ 5073us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:47.839364 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.840101 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.841018 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.841218 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.841328 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.842753 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:47.842756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:47.842759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:47.842781 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:47.844007 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.844410 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.844413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.844415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.844492 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:47.844766 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:47.844769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:47.844771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:47.844932 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:47.845060 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.845285 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.845409 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4643,"thread_start_us":86,"threads_started":1}
I20260927 00:44:47.848143 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:47.851099 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:47.868086 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.0.98.2:35233)
0927 00:44:47.863477 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.863488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.863511 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:47.863633 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.863639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.863864 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.864031 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:47.864237 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.865274 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.866758 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.867500 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867596 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.867721 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.867849 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.867947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":98,"threads_started":1}
I20260927 00:44:47.868072 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35233 (local address 127.0.0.1:36886)
0927 00:44:47.862645 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.862659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.863233 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.864014 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.864017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.864034 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:47.864112 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.864119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.865389 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.865392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.866632 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.866642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.867616 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.867705 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.867731 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.867804 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.867875 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260927 00:44:47.868076 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42557 (local address 127.0.0.1:43468)
0927 00:44:47.862328 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.862345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.862970 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.863610 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.863615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.863636 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:47.863733 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.863739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.865077 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.865080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.866447 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.866458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.867660 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.867763 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.867787 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.867856 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.867924 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":143,"thread_start_us":137,"threads_started":1}
I20260927 00:44:47.868395 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.98.3:42557)
0927 00:44:47.862759 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.862772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.862797 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:44:47.863148 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.863156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.863388 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.863554 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:47.863846 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.864950 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.866546 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.867327 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867445 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.867597 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.868139 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.868242 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":118,"threads_started":1}
I20260927 00:44:47.869549 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36323 (local address 127.0.0.1:57696)
0927 00:44:47.864242 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.864254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.864364 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.864739 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.864742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.864766 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:47.864868 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.864874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.866189 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.866192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.867104 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.867111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.869147 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.869151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.869255 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.869274 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.869367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.869433 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"mutex_wait_us":23,"thread_start_us":74,"threads_started":1}
I20260927 00:44:47.869642 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.862936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.98.1:36323)
0927 00:44:47.863372 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.863430 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.864350 (+ 920us) server_negotiation.cc:407] Connection header received
0927 00:44:47.864467 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.864472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.864611 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.864755 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:47.864955 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.866070 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867218 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.867873 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.867972 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.868101 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.869399 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.869491 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":105,"threads_started":1}
I20260927 00:44:47.871055 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea71aa7473d47b19f57f4da07525ede (DEFAULT_TABLE table=test-table [id=afef24ced04749c5af70966c0b624086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:47.871055 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea71aa7473d47b19f57f4da07525ede (DEFAULT_TABLE table=test-table [id=afef24ced04749c5af70966c0b624086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:47.872128 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea71aa7473d47b19f57f4da07525ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:47.872128 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea71aa7473d47b19f57f4da07525ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:47.872123 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea71aa7473d47b19f57f4da07525ede (DEFAULT_TABLE table=test-table [id=afef24ced04749c5af70966c0b624086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:47.872988 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea71aa7473d47b19f57f4da07525ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:47.879479 7767 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Bootstrap starting.
I20260927 00:44:47.880564 7768 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Bootstrap starting.
I20260927 00:44:47.882079 7769 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Bootstrap starting.
I20260927 00:44:47.882091 7767 tablet_bootstrap.cc:654] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:47.882740 7768 tablet_bootstrap.cc:654] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:47.882988 7767 log.cc:824] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:47.883535 7768 log.cc:824] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:47.884120 7769 tablet_bootstrap.cc:654] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:47.884752 7769 log.cc:824] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:47.885254 7767 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: No bootstrap required, opened a new log
I20260927 00:44:47.885566 7768 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: No bootstrap required, opened a new log
I20260927 00:44:47.885730 7768 ts_tablet_manager.cc:1397] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:47.885730 7767 ts_tablet_manager.cc:1397] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:44:47.886786 7769 tablet_bootstrap.cc:492] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: No bootstrap required, opened a new log
I20260927 00:44:47.887036 7769 ts_tablet_manager.cc:1397] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:47.890849 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.885795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.0.1:48936)
0927 00:44:47.885903 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:47.885922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:47.886010 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:47.886550 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:47.886553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:47.886573 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:47.886677 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.886683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.887730 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.887734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.888924 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:47.888935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.890367 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.890373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:47.890516 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.890547 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:44:47.890631 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:44:47.890715 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 00:44:47.891323 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:47.886162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.98.60:36059)
0927 00:44:47.886285 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:47.886289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:47.886304 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:47.886357 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:47.886360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:47.886417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:47.886514 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:47.886816 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.887596 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.889300 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:47.889997 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:47.890109 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:47.890293 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:47.891130 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 00:44:47.891213 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:47.891630 7768 raft_consensus.cc:348] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.891968 7768 raft_consensus.cc:374] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:47.891809 7767 raft_consensus.cc:348] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.892024 7768 raft_consensus.cc:729] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bcfb9f41f34461a8dedc7c1ef6073a, State: Initialized, Role: FOLLOWER
I20260927 00:44:47.892033 7767 raft_consensus.cc:374] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:47.892086 7767 raft_consensus.cc:729] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e867ec41ee449cbf454c7595b71a49, State: Initialized, Role: FOLLOWER
I20260927 00:44:47.892102 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 00:44:47.891799 7769 raft_consensus.cc:348] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.892351 7769 raft_consensus.cc:374] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:47.892447 7769 raft_consensus.cc:729] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad31d43949c4bfea794419be955e623, State: Initialized, Role: FOLLOWER
I20260927 00:44:47.892534 7768 consensus_queue.cc:260] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.892534 7767 consensus_queue.cc:260] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.893016 7769 consensus_queue.cc:260] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:47.893530 7767 ts_tablet_manager.cc:1428] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:44:47.893579 7768 ts_tablet_manager.cc:1428] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260927 00:44:47.894191 7769 ts_tablet_manager.cc:1428] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:44:47.894667 7602 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
I20260927 00:44:47.894670 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f08d97f85d42b29dad35abbe1ab14d. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:47.894825 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: LEARNER
I20260927 00:44:47.895072 7739 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
I20260927 00:44:47.898604 7465 heartbeater.cc:503] Master 127.0.98.62:33789 was elected leader, sending a full tablet report...
W20260927 00:44:47.944312 7468 tablet.cc:2369] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:48.068419 7742 tablet.cc:2369] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:48.132794 7605 tablet.cc:2369] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:48.211431 7775 raft_consensus.cc:482] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:48.211678 7775 raft_consensus.cc:504] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:48.212934 7775 leader_election.cc:288] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad31d43949c4bfea794419be955e623 (127.0.98.3:42557), 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:48.216758 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.213343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37163 (local address 127.0.98.3:42557)
0927 00:44:48.213476 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.213480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.213496 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:44:48.213659 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.213664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.213729 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.213850 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:48.214109 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.214620 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.215790 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.216183 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.216300 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.216379 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.216536 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.216611 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:44:48.216759 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.213193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42557 (local address 127.0.98.1:37163)
0927 00:44:48.213424 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:48.213441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:48.213543 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:48.213863 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:48.213866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:48.213885 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:48.213986 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:48.213992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.214732 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.214736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:48.215656 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:48.215665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.216300 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.216303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:48.216400 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.216433 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:44:48.216512 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:44:48.216581 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 00:44:48.216908 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.213687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37767 (local address 127.0.98.2:35233)
0927 00:44:48.213845 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.213850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.213864 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:48.214006 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.214009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.214049 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.214129 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:48.214348 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.214830 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.215947 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.216392 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.216520 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.216579 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.216720 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.216791 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:44:48.217115 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.213562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35233 (local address 127.0.98.1:37767)
0927 00:44:48.213756 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:48.213788 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:48.213867 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:48.214139 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:48.214142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:48.214157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:48.214248 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:48.214253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.214941 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.214945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:48.215819 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:48.215831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.216530 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.216533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:48.216607 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.216625 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:44:48.216692 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:44:48.216756 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260927 00:44:48.217525 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea71aa7473d47b19f57f4da07525ede" candidate_uuid: "17e867ec41ee449cbf454c7595b71a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad31d43949c4bfea794419be955e623" is_pre_election: true
I20260927 00:44:48.217718 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea71aa7473d47b19f57f4da07525ede" candidate_uuid: "17e867ec41ee449cbf454c7595b71a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" is_pre_election: true
I20260927 00:44:48.217870 7689 raft_consensus.cc:2454] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e867ec41ee449cbf454c7595b71a49 in term 0.
I20260927 00:44:48.217998 7556 raft_consensus.cc:2454] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e867ec41ee449cbf454c7595b71a49 in term 0.
I20260927 00:44:48.218340 7354 leader_election.cc:302] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e867ec41ee449cbf454c7595b71a49, aad31d43949c4bfea794419be955e623; no voters:
I20260927 00:44:48.218586 7775 raft_consensus.cc:2781] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:48.218683 7775 raft_consensus.cc:482] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:48.218767 7775 raft_consensus.cc:3037] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:48.219794 7775 raft_consensus.cc:504] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:48.220120 7775 leader_election.cc:288] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [CANDIDATE]: Term 1 election: Requested vote from peers aad31d43949c4bfea794419be955e623 (127.0.98.3:42557), 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233)
I20260927 00:44:48.220417 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea71aa7473d47b19f57f4da07525ede" candidate_uuid: "17e867ec41ee449cbf454c7595b71a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad31d43949c4bfea794419be955e623"
I20260927 00:44:48.220508 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea71aa7473d47b19f57f4da07525ede" candidate_uuid: "17e867ec41ee449cbf454c7595b71a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a"
I20260927 00:44:48.220589 7689 raft_consensus.cc:3037] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:48.220645 7556 raft_consensus.cc:3037] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:48.221591 7689 raft_consensus.cc:2454] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e867ec41ee449cbf454c7595b71a49 in term 1.
I20260927 00:44:48.221597 7556 raft_consensus.cc:2454] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e867ec41ee449cbf454c7595b71a49 in term 1.
I20260927 00:44:48.221870 7354 leader_election.cc:302] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e867ec41ee449cbf454c7595b71a49, 38bcfb9f41f34461a8dedc7c1ef6073a; no voters:
I20260927 00:44:48.222051 7775 raft_consensus.cc:2781] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:48.222249 7775 raft_consensus.cc:686] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 1 LEADER]: Becoming Leader. State: Replica: 17e867ec41ee449cbf454c7595b71a49, State: Running, Role: LEADER
I20260927 00:44:48.222568 7775 consensus_queue.cc:237] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:48.225848 6501 catalog_manager.cc:5495] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "17e867ec41ee449cbf454c7595b71a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } health_report { overall_health: HEALTHY } } }
I20260927 00:44:48.227962 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: true } } }
I20260927 00:44:48.228058 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:48.230024 7785 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: attempting to promote NON_VOTER e1e622b6712d43d6a7e464e2fc21309d to VOTER
I20260927 00:44:48.230315 7785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:48.231559 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 LEARNER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 00:44:48.231920 7785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 00:44:48.232044 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Refusing update from remote peer 88f08d97f85d42b29dad35abbe1ab14d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 00:44:48.232357 7785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 00:44:48.233439 7629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.233870 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.234226 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.234895 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.235008 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:48.235173 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.235231 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:48.235337 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.235404 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:48.235771 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.235746 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f08d97f85d42b29dad35abbe1ab14d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:48.235862 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:48.235862 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:48.323930 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.320398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.0.98.1:36323)
0927 00:44:48.320547 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.320554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.320569 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:48.320650 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.320652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.320708 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.320827 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:48.321071 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.321580 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.322462 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.322681 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.322795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.323099 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:48.323523 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:48.323668 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.323706 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.323782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:44:48.329895 7556 raft_consensus.cc:1261] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Refusing update from remote peer 17e867ec41ee449cbf454c7595b71a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:48.329882 7689 raft_consensus.cc:1261] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Refusing update from remote peer 17e867ec41ee449cbf454c7595b71a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:48.330641 7783 consensus_queue.cc:1035] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:48.330801 7775 consensus_queue.cc:1035] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:48.339298 7789 mvcc.cc:205] Tried to move back new op lower bound from 7333764662591836160 to 7333764662162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7333764662591836160}]
I20260927 00:44:48.340544 7790 mvcc.cc:205] Tried to move back new op lower bound from 7333764662591836160 to 7333764662162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7333764662591836160}]
I20260927 00:44:48.341022 7791 mvcc.cc:205] Tried to move back new op lower bound from 7333764662591836160 to 7333764662162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7333764662591836160}]
I20260927 00:44:48.352012 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:48.352183 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:48.356573 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f08d97f85d42b29dad35abbe1ab14d: Sending DeleteTablet for 3 replicas of tablet 4ea71aa7473d47b19f57f4da07525ede
I20260927 00:44:48.357507 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea71aa7473d47b19f57f4da07525ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260927 00:44:48.357692 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea71aa7473d47b19f57f4da07525ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36886
I20260927 00:44:48.357734 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea71aa7473d47b19f57f4da07525ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43468
I20260927 00:44:48.358923 7801 tablet_replica.cc:330] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: stopping tablet replica
I20260927 00:44:48.360055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6470
I20260927 00:44:48.363304 7800 tablet_replica.cc:330] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: stopping tablet replica
I20260927 00:44:48.363620 7800 raft_consensus.cc:2229] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:48.363953 7800 raft_consensus.cc:2258] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:48.365603 7800 ts_tablet_manager.cc:1905] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:48.368790 7800 ts_tablet_manager.cc:1918] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:48.368880 7800 log.cc:1196] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/4ea71aa7473d47b19f57f4da07525ede
I20260927 00:44:48.369189 7800 ts_tablet_manager.cc:1939] T 4ea71aa7473d47b19f57f4da07525ede P aad31d43949c4bfea794419be955e623: Deleting consensus metadata
I20260927 00:44:48.371815 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.359860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.98.61:39033)
0927 00:44:48.362810 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.362817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.362832 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:48.362871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.362877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.362944 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.363029 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:48.363911 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.364768 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.365965 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.366313 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.366409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.368198 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:48.368202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:48.368205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:48.368231 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:48.370040 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.370511 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.370514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.370516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.370605 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.370901 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.370907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.370912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.371161 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:48.371297 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.371526 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.371648 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2853,"thread_start_us":75,"threads_started":1}
W20260927 00:44:48.372169 7628 connection.cc:441] server connection from 127.0.0.1:43468 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43468 (request call id 1) could send its response
I20260927 00:44:48.372897 7801 raft_consensus.cc:2229] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:48.373700 7801 raft_consensus.cc:2258] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:48.374007 7802 tablet_replica.cc:330] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: stopping tablet replica
I20260927 00:44:48.374320 7802 raft_consensus.cc:2229] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:48.374694 7802 raft_consensus.cc:2258] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:48.376915 7801 ts_tablet_manager.cc:1905] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:48.379508 7801 ts_tablet_manager.cc:1918] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:48.379590 7801 log.cc:1196] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/4ea71aa7473d47b19f57f4da07525ede
I20260927 00:44:48.379891 7801 ts_tablet_manager.cc:1939] T 4ea71aa7473d47b19f57f4da07525ede P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting consensus metadata
I20260927 00:44:48.380896 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.359870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.0.98.60:36059)
0927 00:44:48.359999 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.360004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.371910 (+ 11906us) server_negotiation.cc:407] Connection header received
0927 00:44:48.371952 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.371955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.372003 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.372078 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:48.373033 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.373834 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.374898 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.375235 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.375336 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.376962 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:48.376968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:48.376972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:48.376997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:48.378873 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.379379 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.379383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.379385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.379465 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.379721 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.379724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.379726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.379949 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:48.380074 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.380614 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.380727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:44:48.380934 7492 connection.cc:441] server connection from 127.0.0.1:36886 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36886 (request call id 1) could send its response
I20260927 00:44:48.382387 7802 ts_tablet_manager.cc:1905] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:48.384917 7802 ts_tablet_manager.cc:1918] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:48.385018 7802 log.cc:1196] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/4ea71aa7473d47b19f57f4da07525ede
I20260927 00:44:48.385329 7802 ts_tablet_manager.cc:1939] T 4ea71aa7473d47b19f57f4da07525ede P 17e867ec41ee449cbf454c7595b71a49: Deleting consensus metadata
W20260927 00:44:48.386380 7353 connection.cc:441] server connection from 127.0.0.1:57696 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57696 (request call id 1) could send its response
I20260927 00:44:48.389223 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.373081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.98.61:39033)
0927 00:44:48.373387 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.373393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.375901 (+ 2508us) server_negotiation.cc:407] Connection header received
0927 00:44:48.376084 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.376087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.376137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.376189 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:48.379021 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.379811 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.383183 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.383536 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.383611 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.385553 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:48.385558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:48.385560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:48.385591 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:48.387439 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.387926 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.387929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.387931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.388019 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.388315 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.388321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.388324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.388559 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:48.388689 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.388933 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.389065 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260927 00:44:48.390295 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:48.373446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.98.60:36059)
0927 00:44:48.375667 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:48.375672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:48.376508 (+ 836us) server_negotiation.cc:407] Connection header received
0927 00:44:48.377413 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:48.377417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:48.377464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:48.377533 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:48.381038 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.382213 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.383137 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:48.383612 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:48.383712 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:48.385679 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:48.385682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:48.385687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:48.385712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:48.387881 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.388364 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.388369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.388372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.388443 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:48.388752 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:48.388757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:48.388760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:48.388963 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:48.389084 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:48.390009 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 00:44:48.390135 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2142,"thread_start_us":65,"threads_started":1}
I20260927 00:44:49.344393 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.343623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:38611)
0927 00:44:49.343917 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.344006 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":69,"threads_started":1}
W20260927 00:44:49.344676 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.345260 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.344453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:44931)
0927 00:44:49.345096 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.345172 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":66,"threads_started":1}
W20260927 00:44:49.345628 7739 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.348219 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.347823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:46969)
0927 00:44:49.348060 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.348146 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20260927 00:44:49.348517 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33789 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.856909 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88f08d97f85d42b29dad35abbe1ab14d)
I20260927 00:44:49.857178 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:49.858356 7816 leader_election.cc:288] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059)
I20260927 00:44:49.858896 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88f08d97f85d42b29dad35abbe1ab14d)
I20260927 00:44:49.859246 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.858605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59516)
0927 00:44:49.859023 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.859102 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":22,"thread_start_us":62,"threads_started":1}
I20260927 00:44:49.859143 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
W20260927 00:44:49.859566 7089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.859699 7819 leader_election.cc:288] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), ddd7188520aa4c03857953e59e405e29 (127.0.98.61:39033)
I20260927 00:44:49.860395 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.859838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59528)
0927 00:44:49.860198 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.860280 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":61,"threads_started":1}
W20260927 00:44:49.860862 7272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.860921 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.860595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59532)
0927 00:44:49.860806 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.860860 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 00:44:49.861207 7089 leader_election.cc:334] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.861706 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.861480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59544)
0927 00:44:49.861591 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.861642 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:44:49.861917 7272 leader_election.cc:334] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.863534 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.858833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.0.98.60:36059)
0927 00:44:49.859071 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:49.859077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:49.859096 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:44:49.859213 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:49.859216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:49.859267 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:49.859385 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:49.860133 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.860976 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.862509 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.862979 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.863055 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:49.863133 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:49.863341 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:44:49.863417 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260927 00:44:49.863540 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.858605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36059 (local address 127.0.0.1:48970)
0927 00:44:49.858957 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.858975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:49.859099 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:49.859396 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:49.859398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:49.859417 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:49.859994 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:49.860002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.861106 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.861111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:49.862373 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:49.862382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.863084 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.863090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:49.863191 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:49.863221 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:44:49.863296 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:44:49.863415 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":21,"thread_start_us":72,"threads_started":1}
I20260927 00:44:49.864002 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd7188520aa4c03857953e59e405e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1e622b6712d43d6a7e464e2fc21309d" is_pre_election: true
I20260927 00:44:49.864216 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddd7188520aa4c03857953e59e405e29 in term 1.
I20260927 00:44:49.864336 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.859889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39033 (local address 127.0.0.1:39890)
0927 00:44:49.860198 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.860212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:49.860291 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:49.860640 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:49.860643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:49.860660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:49.860866 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:49.860874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.861913 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.861915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:49.863001 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:49.863009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.863916 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.863920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:49.864000 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:49.864031 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:44:49.864111 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:44:49.864174 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260927 00:44:49.864370 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.859968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.98.61:39033)
0927 00:44:49.860237 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:49.860245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:49.860259 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:49.860404 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:49.860410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:49.860460 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:49.860540 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:49.860997 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.861729 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.863139 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:49.863805 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:49.863899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:49.863951 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:49.864149 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:44:49.864217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260927 00:44:49.864641 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1e622b6712d43d6a7e464e2fc21309d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ddd7188520aa4c03857953e59e405e29" is_pre_election: true
I20260927 00:44:49.864656 7088 leader_election.cc:302] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddd7188520aa4c03857953e59e405e29, e1e622b6712d43d6a7e464e2fc21309d; no voters: 88f08d97f85d42b29dad35abbe1ab14d
I20260927 00:44:49.864804 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1e622b6712d43d6a7e464e2fc21309d in term 1.
I20260927 00:44:49.864915 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:44:49.864967 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88f08d97f85d42b29dad35abbe1ab14d)
I20260927 00:44:49.865005 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:44:49.865111 7271 leader_election.cc:302] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddd7188520aa4c03857953e59e405e29, e1e622b6712d43d6a7e464e2fc21309d; no voters: 88f08d97f85d42b29dad35abbe1ab14d
I20260927 00:44:49.865271 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:44:49.865327 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88f08d97f85d42b29dad35abbe1ab14d)
I20260927 00:44:49.865365 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:44:49.866331 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:49.866533 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:49.866744 7816 leader_election.cc:288] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 election: Requested vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059)
I20260927 00:44:49.866856 7819 leader_election.cc:288] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 election: Requested vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), ddd7188520aa4c03857953e59e405e29 (127.0.98.61:39033)
I20260927 00:44:49.867085 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd7188520aa4c03857953e59e405e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1e622b6712d43d6a7e464e2fc21309d"
I20260927 00:44:49.867169 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.866956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59550)
0927 00:44:49.867045 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.867094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 00:44:49.867192 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.866965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59552)
0927 00:44:49.867097 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.867138 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 00:44:49.867261 7296 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ddd7188520aa4c03857953e59e405e29 in current term 2: Already voted for candidate e1e622b6712d43d6a7e464e2fc21309d in this term.
I20260927 00:44:49.867478 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1e622b6712d43d6a7e464e2fc21309d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ddd7188520aa4c03857953e59e405e29"
I20260927 00:44:49.867617 7113 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e1e622b6712d43d6a7e464e2fc21309d in current term 2: Already voted for candidate ddd7188520aa4c03857953e59e405e29 in this term.
I20260927 00:44:49.867831 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.867647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59562)
0927 00:44:49.867726 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.867773 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 00:44:49.867959 7089 leader_election.cc:334] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.867996 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:49.867751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59568)
0927 00:44:49.867894 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:49.867941 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 00:44:49.868021 7089 leader_election.cc:302] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddd7188520aa4c03857953e59e405e29; no voters: 88f08d97f85d42b29dad35abbe1ab14d, e1e622b6712d43d6a7e464e2fc21309d
W20260927 00:44:49.868074 7272 leader_election.cc:334] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:49.868134 7272 leader_election.cc:302] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1e622b6712d43d6a7e464e2fc21309d; no voters: 88f08d97f85d42b29dad35abbe1ab14d, ddd7188520aa4c03857953e59e405e29
I20260927 00:44:49.868153 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:44:49.868340 7819 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:44:50.345862 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:50.345309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:49321)
0927 00:44:50.345638 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:50.345710 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260927 00:44:50.346624 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:50.346068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:48743)
0927 00:44:50.346447 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:50.346518 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":59,"threads_started":1}
I20260927 00:44:50.349390 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:50.348999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:42491)
0927 00:44:50.349270 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:50.349318 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260927 00:44:51.347090 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.346645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:57067)
0927 00:44:51.346920 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.346986 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":125,"threads_started":1}
I20260927 00:44:51.347781 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.347212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:33523)
0927 00:44:51.347620 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.347682 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":80,"threads_started":1}
I20260927 00:44:51.350384 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.350015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:40389)
0927 00:44:51.350263 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.350313 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
I20260927 00:44:51.725449 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:51.725615 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:51.726092 7832 leader_election.cc:288] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059)
I20260927 00:44:51.726563 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd7188520aa4c03857953e59e405e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1e622b6712d43d6a7e464e2fc21309d" is_pre_election: true
I20260927 00:44:51.726732 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.726331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59796)
0927 00:44:51.726555 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.726611 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260927 00:44:51.726755 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddd7188520aa4c03857953e59e405e29 in term 2.
I20260927 00:44:51.727164 7088 leader_election.cc:302] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddd7188520aa4c03857953e59e405e29, e1e622b6712d43d6a7e464e2fc21309d; no voters:
I20260927 00:44:51.727394 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 00:44:51.727455 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:51.727501 7832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:44:51.727615 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.727342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59810)
0927 00:44:51.727434 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.727489 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 00:44:51.727713 7089 leader_election.cc:334] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:51.728600 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:51.728919 7832 leader_election.cc:288] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 election: Requested vote from peers 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789), e1e622b6712d43d6a7e464e2fc21309d (127.0.98.60:36059)
I20260927 00:44:51.729230 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.729044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59816)
0927 00:44:51.729135 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.729183 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 00:44:51.729277 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddd7188520aa4c03857953e59e405e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1e622b6712d43d6a7e464e2fc21309d"
I20260927 00:44:51.729404 7296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:44:51.729869 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.729626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59826)
0927 00:44:51.729727 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.729809 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:44:51.729965 7089 leader_election.cc:334] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
I20260927 00:44:51.730474 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddd7188520aa4c03857953e59e405e29 in term 3.
I20260927 00:44:51.730757 7088 leader_election.cc:302] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddd7188520aa4c03857953e59e405e29, e1e622b6712d43d6a7e464e2fc21309d; no voters: 88f08d97f85d42b29dad35abbe1ab14d
I20260927 00:44:51.730937 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:44:51.731110 7832 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [term 3 LEADER]: Becoming Leader. State: Replica: ddd7188520aa4c03857953e59e405e29, State: Running, Role: LEADER
I20260927 00:44:51.731367 7832 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } }
I20260927 00:44:51.732779 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd7188520aa4c03857953e59e405e29. Latest consensus state: current_term: 3 leader_uuid: "ddd7188520aa4c03857953e59e405e29" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:51.732887 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:51.733155 7837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:51.735347 7837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:51.735692 7837 catalog_manager.cc:1259] Loaded cluster ID: 9446a0a341d04ab4b442b41395102c56
I20260927 00:44:51.735733 7837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:51.736223 7837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:51.736550 7837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Loaded TSK: 0
I20260927 00:44:51.737205 7837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:44:51.768282 7101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:51.771997 7101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:51.775103 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.774840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59836)
0927 00:44:51.774975 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.775035 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:44:51.775206 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [term 3 FOLLOWER]: Refusing update from remote peer ddd7188520aa4c03857953e59e405e29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 00:44:51.775820 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.775564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59848)
0927 00:44:51.775673 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.775729 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 00:44:51.775708 7835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 00:44:51.775976 7089 consensus_peers.cc:487] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 -> Peer 88f08d97f85d42b29dad35abbe1ab14d (127.0.98.62:33789): Couldn't send request to peer 88f08d97f85d42b29dad35abbe1ab14d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:44:51.777173 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd7188520aa4c03857953e59e405e29. Latest consensus state: current_term: 3 leader_uuid: "ddd7188520aa4c03857953e59e405e29" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:51.777344 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:51.778427 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ddd7188520aa4c03857953e59e405e29" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:51.778548 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:51.779260 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ddd7188520aa4c03857953e59e405e29" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88f08d97f85d42b29dad35abbe1ab14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33789 } } peers { permanent_uuid: "ddd7188520aa4c03857953e59e405e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39033 } attrs { promote: false } } peers { permanent_uuid: "e1e622b6712d43d6a7e464e2fc21309d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36059 } attrs { promote: false } } }
I20260927 00:44:51.779352 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e622b6712d43d6a7e464e2fc21309d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:44:51.789458 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.784953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42557 (local address 127.0.0.1:43472)
0927 00:44:51.785193 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.785208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:51.785293 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:51.785707 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:51.785709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:51.785732 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:51.785885 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.785893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.786914 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.786917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.788104 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.788114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.788993 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.788997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.789099 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789125 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:44:51.789195 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:44:51.789307 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":11,"thread_start_us":107,"threads_started":1}
I20260927 00:44:51.789570 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.785182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.0.98.2:35233)
0927 00:44:51.786035 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:51.786039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:51.786054 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:51.786105 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:51.786108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:51.786159 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:51.786279 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:51.786522 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.787227 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.788397 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.788982 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.789111 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789175 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:51.789346 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:44:51.789423 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":77,"threads_started":1}
I20260927 00:44:51.789577 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.784708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35233 (local address 127.0.0.1:36900)
0927 00:44:51.784839 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.784856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:51.784944 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:51.786285 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:51.786288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:51.786305 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:51.786388 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.786394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.787333 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.787338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.788260 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.788272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.789121 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.789124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.789210 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789239 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:44:51.789316 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:44:51.789382 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":48}
I20260927 00:44:51.790102 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.785272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36323 (local address 127.0.0.1:57724)
0927 00:44:51.785475 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.785487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:51.785563 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:51.786417 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:51.786420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:51.786436 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:51.786522 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.786530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.787495 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.787498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.788635 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:51.788643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.789697 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.789701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:51.789814 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789835 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:44:51.789910 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:44:51.789977 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260927 00:44:51.790174 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.785729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.1:36323)
0927 00:44:51.786102 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:51.786107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:51.786119 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:44:51.786166 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:51.786169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:51.786286 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:51.786403 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:51.786640 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.787360 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.788776 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.789403 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.789531 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789592 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:51.789961 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 00:44:51.790046 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":109,"threads_started":1}
I20260927 00:44:51.790261 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.785103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.0.98.3:42557)
0927 00:44:51.785464 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:51.785469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:51.785483 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:51.785562 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:51.785565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:51.785614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:51.785707 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:51.786064 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.786803 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.788242 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:51.788875 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:51.788974 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:51.789037 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:51.790044 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 00:44:51.790126 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":72,"threads_started":1}
I20260927 00:44:51.790382 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 (DEFAULT_TABLE table=test-table [id=6c8506a8af4a4dd780489ca4c9953e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:51.790709 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6475e4f4c8e045bcb6e19067d42d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:51.790853 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 (DEFAULT_TABLE table=test-table [id=6c8506a8af4a4dd780489ca4c9953e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:51.790884 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 (DEFAULT_TABLE table=test-table [id=6c8506a8af4a4dd780489ca4c9953e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:51.791069 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6475e4f4c8e045bcb6e19067d42d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:51.791144 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6475e4f4c8e045bcb6e19067d42d55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:51.793473 7850 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Bootstrap starting.
I20260927 00:44:51.793874 7851 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49: Bootstrap starting.
I20260927 00:44:51.794157 7849 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Bootstrap starting.
I20260927 00:44:51.794610 7850 tablet_bootstrap.cc:654] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:51.794824 7851 tablet_bootstrap.cc:654] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:51.795099 7849 tablet_bootstrap.cc:654] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:51.796270 7851 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49: No bootstrap required, opened a new log
I20260927 00:44:51.796406 7851 ts_tablet_manager.cc:1397] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:51.796434 7849 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: No bootstrap required, opened a new log
I20260927 00:44:51.796500 7849 ts_tablet_manager.cc:1397] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:51.796519 7850 tablet_bootstrap.cc:492] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: No bootstrap required, opened a new log
I20260927 00:44:51.796600 7850 ts_tablet_manager.cc:1397] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:51.796890 7851 raft_consensus.cc:348] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797024 7851 raft_consensus.cc:374] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:51.796964 7849 raft_consensus.cc:348] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797057 7851 raft_consensus.cc:729] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e867ec41ee449cbf454c7595b71a49, State: Initialized, Role: FOLLOWER
I20260927 00:44:51.797062 7849 raft_consensus.cc:374] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:51.796996 7850 raft_consensus.cc:348] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797103 7849 raft_consensus.cc:729] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad31d43949c4bfea794419be955e623, State: Initialized, Role: FOLLOWER
I20260927 00:44:51.797128 7850 raft_consensus.cc:374] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:51.797158 7850 raft_consensus.cc:729] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bcfb9f41f34461a8dedc7c1ef6073a, State: Initialized, Role: FOLLOWER
I20260927 00:44:51.797261 7851 consensus_queue.cc:260] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797261 7849 consensus_queue.cc:260] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797288 7850 consensus_queue.cc:260] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:51.797621 7851 ts_tablet_manager.cc:1428] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:51.797621 7849 ts_tablet_manager.cc:1428] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:51.797641 7850 ts_tablet_manager.cc:1428] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:51.798389 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.797961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:45829)
0927 00:44:51.798239 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.798304 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 00:44:51.798731 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.798079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:50653)
0927 00:44:51.798604 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.798660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260927 00:44:51.799369 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.799130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:48627)
0927 00:44:51.799260 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.799307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 00:44:51.799701 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.798875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:58813)
0927 00:44:51.799574 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.799631 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624}
I20260927 00:44:51.800724 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.800493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:59957)
0927 00:44:51.800623 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.800663 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 00:44:51.801126 7741 heartbeater.cc:503] Master 127.0.98.61:39033 was elected leader, sending a full tablet report...
I20260927 00:44:51.801218 7604 heartbeater.cc:503] Master 127.0.98.61:39033 was elected leader, sending a full tablet report...
I20260927 00:44:51.802274 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:51.802004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:60457)
0927 00:44:51.802143 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:51.802192 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 00:44:51.804177 7467 heartbeater.cc:503] Master 127.0.98.61:39033 was elected leader, sending a full tablet report...
I20260927 00:44:52.134804 7857 raft_consensus.cc:482] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:52.134958 7857 raft_consensus.cc:504] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:52.136184 7857 leader_election.cc:288] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad31d43949c4bfea794419be955e623 (127.0.98.3:42557), 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:52.141160 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.136626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34389 (local address 127.0.98.3:42557)
0927 00:44:52.136765 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:52.136772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:52.136794 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:44:52.136952 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:52.136955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:52.137008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:52.137102 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:52.137439 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.138295 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.139844 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.140473 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.140589 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:52.140651 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:52.140958 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 00:44:52.141033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:44:52.141160 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.137004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38433 (local address 127.0.98.1:36323)
0927 00:44:52.137123 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:52.137177 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:52.137255 (+ 78us) server_negotiation.cc:407] Connection header received
0927 00:44:52.137357 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:52.137359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:52.137425 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:52.137506 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:52.137731 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.138499 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.139844 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.140473 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.140589 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:52.140650 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:52.140958 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:44:52.141033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:44:52.141183 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.136549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42557 (local address 127.0.98.2:34389)
0927 00:44:52.136682 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.136700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:52.136824 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:52.137148 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:52.137151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:52.137171 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:52.137325 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:52.137334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.138397 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.138400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:52.139682 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:52.139693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.140636 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.140640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:52.140799 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:52.140847 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:44:52.140923 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:44:52.141022 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":73}
I20260927 00:44:52.141192 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.136836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36323 (local address 127.0.98.2:38433)
0927 00:44:52.137162 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.137175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:52.137261 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:52.137525 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:52.137527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:52.137542 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:52.137620 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:52.137624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.138597 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.138600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:52.139702 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:52.139708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.140636 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.140640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:52.140794 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:52.140842 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:44:52.140923 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:44:52.141022 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":31,"thread_start_us":141,"threads_started":1}
I20260927 00:44:52.141722 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6475e4f4c8e045bcb6e19067d42d55e5" candidate_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad31d43949c4bfea794419be955e623" is_pre_election: true
I20260927 00:44:52.141894 7689 raft_consensus.cc:2454] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bcfb9f41f34461a8dedc7c1ef6073a in term 0.
I20260927 00:44:52.142098 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6475e4f4c8e045bcb6e19067d42d55e5" candidate_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e867ec41ee449cbf454c7595b71a49" is_pre_election: true
I20260927 00:44:52.142275 7491 leader_election.cc:302] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bcfb9f41f34461a8dedc7c1ef6073a, aad31d43949c4bfea794419be955e623; no voters:
I20260927 00:44:52.142422 7419 raft_consensus.cc:2454] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bcfb9f41f34461a8dedc7c1ef6073a in term 0.
I20260927 00:44:52.142561 7857 raft_consensus.cc:2781] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:52.142627 7857 raft_consensus.cc:482] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:52.142724 7857 raft_consensus.cc:3037] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:52.143971 7857 raft_consensus.cc:504] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:52.144256 7857 leader_election.cc:288] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [CANDIDATE]: Term 1 election: Requested vote from peers aad31d43949c4bfea794419be955e623 (127.0.98.3:42557), 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323)
I20260927 00:44:52.144598 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6475e4f4c8e045bcb6e19067d42d55e5" candidate_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad31d43949c4bfea794419be955e623"
I20260927 00:44:52.144735 7689 raft_consensus.cc:3037] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:52.144769 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6475e4f4c8e045bcb6e19067d42d55e5" candidate_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e867ec41ee449cbf454c7595b71a49"
I20260927 00:44:52.144891 7419 raft_consensus.cc:3037] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:52.145653 7689 raft_consensus.cc:2454] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bcfb9f41f34461a8dedc7c1ef6073a in term 1.
I20260927 00:44:52.145928 7491 leader_election.cc:302] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bcfb9f41f34461a8dedc7c1ef6073a, aad31d43949c4bfea794419be955e623; no voters:
I20260927 00:44:52.145924 7419 raft_consensus.cc:2454] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bcfb9f41f34461a8dedc7c1ef6073a in term 1.
I20260927 00:44:52.146087 7857 raft_consensus.cc:2781] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:52.146313 7857 raft_consensus.cc:686] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 LEADER]: Becoming Leader. State: Replica: 38bcfb9f41f34461a8dedc7c1ef6073a, State: Running, Role: LEADER
I20260927 00:44:52.146782 7857 consensus_queue.cc:237] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } }
I20260927 00:44:52.147044 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.146788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:48855)
0927 00:44:52.146908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.146966 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:44:52.149111 7102 catalog_manager.cc:5495] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a reported cstate change: term changed from 0 to 1, leader changed from <none> to 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38bcfb9f41f34461a8dedc7c1ef6073a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:52.239837 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.236414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.0.98.2:35233)
0927 00:44:52.236540 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:52.236543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:52.236556 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:52.236629 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:52.236631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:52.236685 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:52.236794 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:52.237004 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.237507 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.238410 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:52.238665 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:52.238754 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:52.239002 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:52.239489 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:52.239596 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:52.239671 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 00:44:52.239726 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:44:52.242954 7689 raft_consensus.cc:1261] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Refusing update from remote peer 38bcfb9f41f34461a8dedc7c1ef6073a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:52.243181 7419 raft_consensus.cc:1261] T 6475e4f4c8e045bcb6e19067d42d55e5 P 17e867ec41ee449cbf454c7595b71a49 [term 1 FOLLOWER]: Refusing update from remote peer 38bcfb9f41f34461a8dedc7c1ef6073a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:52.243691 7857 consensus_queue.cc:1035] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad31d43949c4bfea794419be955e623" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:52.243896 7859 consensus_queue.cc:1035] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e867ec41ee449cbf454c7595b71a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:52.246225 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.245955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:45931)
0927 00:44:52.246080 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.246144 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:44:52.246698 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.246454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:43087)
0927 00:44:52.246564 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.246622 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:44:52.248916 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.248264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:39321)
0927 00:44:52.248766 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.248820 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
I20260927 00:44:52.249739 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.249455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.3:45161)
0927 00:44:52.249613 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.249660 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 00:44:52.250540 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.250146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.2:32865)
0927 00:44:52.250430 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.250478 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260927 00:44:52.251289 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.251039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.98.1:33073)
0927 00:44:52.251175 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.251224 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:44:52.259626 7103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:52.259788 7103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:52.263335 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.263005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59860)
0927 00:44:52.263195 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.263252 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 00:44:52.263789 7103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddd7188520aa4c03857953e59e405e29: Sending DeleteTablet for 3 replicas of tablet 6475e4f4c8e045bcb6e19067d42d55e5
I20260927 00:44:52.264192 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.263936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33789 (local address 127.0.0.1:59862)
0927 00:44:52.264085 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.264130 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 00:44:52.264858 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:52 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36900
I20260927 00:44:52.264938 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:52 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43472
I20260927 00:44:52.265072 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 6475e4f4c8e045bcb6e19067d42d55e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:52 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57724
I20260927 00:44:52.265244 7872 tablet_replica.cc:330] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: stopping tablet replica
I20260927 00:44:52.265241 7873 tablet_replica.cc:330] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: stopping tablet replica
I20260927 00:44:52.265353 7872 raft_consensus.cc:2229] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:52.265353 7873 raft_consensus.cc:2229] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:52.265482 7873 raft_consensus.cc:2258] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:52.265687 7872 raft_consensus.cc:2258] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:52.266588 7873 ts_tablet_manager.cc:1905] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:52.266685 7872 ts_tablet_manager.cc:1905] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:52.266721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7247
I20260927 00:44:52.269217 7873 ts_tablet_manager.cc:1918] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:52.269224 7872 ts_tablet_manager.cc:1918] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:52.269289 7872 log.cc:1196] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/6475e4f4c8e045bcb6e19067d42d55e5
I20260927 00:44:52.269289 7873 log.cc:1196] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/6475e4f4c8e045bcb6e19067d42d55e5
I20260927 00:44:52.269567 7873 ts_tablet_manager.cc:1939] T 6475e4f4c8e045bcb6e19067d42d55e5 P aad31d43949c4bfea794419be955e623: Deleting consensus metadata
I20260927 00:44:52.269567 7872 ts_tablet_manager.cc:1939] T 6475e4f4c8e045bcb6e19067d42d55e5 P 38bcfb9f41f34461a8dedc7c1ef6073a: Deleting consensus metadata
I20260927 00:44:52.270383 7089 catalog_manager.cc:4841] TS aad31d43949c4bfea794419be955e623 (127.0.98.3:42557): tablet 6475e4f4c8e045bcb6e19067d42d55e5 (table test-table [id=6c8506a8af4a4dd780489ca4c9953e16]) successfully deleted
I20260927 00:44:52.271070 7089 catalog_manager.cc:4841] TS 38bcfb9f41f34461a8dedc7c1ef6073a (127.0.98.2:35233): tablet 6475e4f4c8e045bcb6e19067d42d55e5 (table test-table [id=6c8506a8af4a4dd780489ca4c9953e16]) successfully deleted
I20260927 00:44:52.276582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7473
I20260927 00:44:52.276630 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:52.276378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36323 (local address 127.0.0.1:57738)
0927 00:44:52.276510 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:52.276563 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:44:52.277967 7089 catalog_manager.cc:4584] TS 17e867ec41ee449cbf454c7595b71a49 (127.0.98.1:36323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6475e4f4c8e045bcb6e19067d42d55e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36323: connect: Connection refused (error 111)
I20260927 00:44:52.282712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7609
I20260927 00:44:52.288659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 6966
I20260927 00:44:52.294888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7069
2026-09-27T00:44:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7713 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:52 dist-test-slave-l5sb krb5kdc[7877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:52 dist-test-slave-l5sb krb5kdc[7877](info): set up 2 sockets
Sep 27 00:44:52 dist-test-slave-l5sb krb5kdc[7877](info): commencing operation
krb5kdc: starting...
W20260927 00:44:53.330426 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:53 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:44:53.449038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:53.555382 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:53.555660 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:53.555728 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:53.559679 7890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:53.559764 7890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:44:53.559795 7890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:53.559837 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:53.559876 7890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:53.559912 7890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:53.565061 7890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:53.566087 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:53.567073 7890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:53.575114 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:53.575148 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:53.575102 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:53.575320 7890 server_base.cc:1034] running on GCE node
I20260927 00:44:53.575946 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:53.577003 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:53.578156 7890 hybrid_clock.cc:648] HybridClock initialized: now 1790469893578146 us; error 27 us; skew 500 ppm
Sep 27 00:44:53 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:53.580832 7890 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:44:53.581683 7890 webserver.cc:460] Webserver started at http://127.0.98.62:38111/ using document root <none> and password file <none>
I20260927 00:44:53.582244 7890 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:53.582330 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:53.582544 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:53.584470 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "d214220287a34666a659e326ab1491a0"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "559a2b1fe128c7ca234fbbb402082a97"
server_key_iv: "c7b70566aaf389588c6e5167965cfabf"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.585021 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "d214220287a34666a659e326ab1491a0"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "559a2b1fe128c7ca234fbbb402082a97"
server_key_iv: "c7b70566aaf389588c6e5167965cfabf"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.588563 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:44:53.590942 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:53.592118 7890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:53.592255 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "d214220287a34666a659e326ab1491a0"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "559a2b1fe128c7ca234fbbb402082a97"
server_key_iv: "c7b70566aaf389588c6e5167965cfabf"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.592361 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:53.620862 7890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:53.623704 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:53.623904 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:53.631450 7890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44681
I20260927 00:44:53.631471 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44681 every 8 connection(s)
I20260927 00:44:53.632397 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:53.634850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7890
I20260927 00:44:53.634999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:53.635272 7968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:53.635322 392 external_mini_cluster.cc:1428] Setting key 7fb00135cb02ede00965919e282200bd
I20260927 00:44:53.641404 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Bootstrap starting.
I20260927 00:44:53.643886 7968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Neither blocks nor log segments found. Creating new log.
Sep 27 00:44:53 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:53.644794 7968 log.cc:824] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:53.647075 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: No bootstrap required, opened a new log
I20260927 00:44:53.650329 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:53.636691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.98.62:44681)
0927 00:44:53.637088 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:53.637102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:53.637134 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:44:53.638019 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:53.638037 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:53.638374 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:53.638698 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:53.639513 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.640471 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.641312 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.641875 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.642036 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:53.644705 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:53.644710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:53.644722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:53.644747 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:53.647372 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.647794 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.647799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.647805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.647944 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.648207 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.648210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.648212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.648673 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:53.648847 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:53.649106 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:44:53.649392 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260927 00:44:53.650924 7968 raft_consensus.cc:348] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260927 00:44:53.651172 7968 raft_consensus.cc:374] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:53.651283 7968 raft_consensus.cc:729] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d214220287a34666a659e326ab1491a0, State: Initialized, Role: FOLLOWER
I20260927 00:44:53.651746 7968 consensus_queue.cc:260] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260927 00:44:53.651896 7968 raft_consensus.cc:388] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:53.651958 7968 raft_consensus.cc:482] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:53.652045 7968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:53.653033 7968 raft_consensus.cc:504] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260927 00:44:53.653304 7968 leader_election.cc:302] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d214220287a34666a659e326ab1491a0; no voters:
I20260927 00:44:53.653550 7968 leader_election.cc:288] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:53.653681 7973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:53.653940 7973 raft_consensus.cc:686] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [term 1 LEADER]: Becoming Leader. State: Replica: d214220287a34666a659e326ab1491a0, State: Running, Role: LEADER
I20260927 00:44:53.654287 7973 consensus_queue.cc:237] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260927 00:44:53.654678 7968 sys_catalog.cc:564] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:53.655966 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d214220287a34666a659e326ab1491a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } } }
I20260927 00:44:53.655972 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d214220287a34666a659e326ab1491a0. Latest consensus state: current_term: 1 leader_uuid: "d214220287a34666a659e326ab1491a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d214220287a34666a659e326ab1491a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } } }
I20260927 00:44:53.656132 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:53.656168 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:53.656526 7985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:53.659627 7985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:53.664981 7985 catalog_manager.cc:1347] Generated new cluster ID: 85fc5c32e80445fb804a4f5136486c46
I20260927 00:44:53.665046 7985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:53.680069 7985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:53.681226 7985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:53.697504 7985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Generated new TSK 0
I20260927 00:44:53.698372 7985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:44:53.728406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44681
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:53.838459 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:53.838694 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:53.838737 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:53.842718 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:53.842768 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:53.842808 7994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:53.842839 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:53.847995 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:53.848924 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:53.849944 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:53.859354 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:53.859404 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:53.859493 7994 server_base.cc:1034] running on GCE node
W20260927 00:44:53.859663 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:53.860029 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:53.860622 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:53.861824 7994 hybrid_clock.cc:648] HybridClock initialized: now 1790469893861801 us; error 29 us; skew 500 ppm
Sep 27 00:44:53 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:53.864387 7994 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:44:53.865154 7994 webserver.cc:460] Webserver started at http://127.0.98.1:41621/ using document root <none> and password file <none>
I20260927 00:44:53.865657 7994 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:53.865734 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:53.865955 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:53.867868 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "29716c38851c4adaa4842959042bd564"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "9a379bb9682dfaf08ea258fb0bfcf71c"
server_key_iv: "2b670fc7f5182277709ddd89cc11e1c0"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.868355 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29716c38851c4adaa4842959042bd564"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "9a379bb9682dfaf08ea258fb0bfcf71c"
server_key_iv: "2b670fc7f5182277709ddd89cc11e1c0"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.871919 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:44:53.874217 8009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:53.875494 7994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:53.875600 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "29716c38851c4adaa4842959042bd564"
format_stamp: "Formatted at 2026-09-27 00:44:53 on dist-test-slave-l5sb"
server_key: "9a379bb9682dfaf08ea258fb0bfcf71c"
server_key_iv: "2b670fc7f5182277709ddd89cc11e1c0"
server_key_version: "encryptionkey@0"
I20260927 00:44:53.875717 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:53.896590 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:53.899868 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:53.900072 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:53.901474 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:53.901553 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:53.901623 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:53.901680 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:53 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:53.912616 7994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44447
I20260927 00:44:53.912645 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44447 every 8 connection(s)
I20260927 00:44:53.913527 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:53.914193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 7994
I20260927 00:44:53.914326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:53.914675 392 external_mini_cluster.cc:1428] Setting key b01db1934207d0daa48872d121d6dd36
I20260927 00:44:53.915978 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:53.902328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33129 (local address 127.0.98.62:44681)
0927 00:44:53.902447 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:53.902451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:53.903469 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 00:44:53.904423 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:53.904427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:53.904481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:53.904539 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:53.906241 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.906791 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.907913 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.908209 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.908322 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:53.911881 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:53.911884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:53.911887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:53.911914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:53.913529 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.914267 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.914272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.914276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.914374 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.914760 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.914766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.914771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.914974 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:53.915092 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:53.915678 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 00:44:53.915784 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:53.916908 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:53.902859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.1:33129)
0927 00:44:53.903327 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:53.903369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:53.904160 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:53.904975 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:53.904986 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:53.905472 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:53.906093 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:53.906108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.906916 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.906920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:53.907789 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:53.907797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.908374 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.908378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:53.908539 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:53.909146 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:53.909170 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:53.911716 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:53.913671 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:53.913677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:53.913692 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:53.914102 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:53.914464 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:53.914467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:53.914469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:53.914658 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:53.915109 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:53.915115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:53.915422 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 00:44:53.915612 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 00:44:53.915928 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":138,"threads_started":1}
I20260927 00:44:53.918234 8125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44681
I20260927 00:44:53.918394 8125 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:53.918864 8125 heartbeater.cc:511] Master 127.0.98.62:44681 requested a full tablet report, sending...
I20260927 00:44:53.920419 7921 ts_manager.cc:194] Registered new tserver with Master: 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447)
I20260927 00:44:53.921588 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33129
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:53.928542 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:53.920630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50451 (local address 127.0.98.62:44681)
0927 00:44:53.920769 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:53.920774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:53.921122 (+ 348us) server_negotiation.cc:407] Connection header received
0927 00:44:53.921310 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:53.921316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:53.921371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:53.921427 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:53.922163 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.923031 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.924335 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.924543 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.924623 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:53.925753 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:53.925756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:53.925758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:53.925818 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:53.927026 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.927402 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.927405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.927406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.927475 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:53.927747 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:53.927752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:53.927753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:53.927922 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:53.928029 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:53.928284 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:44:53.928400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:53.928551 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:53.920754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.1:50451)
0927 00:44:53.921042 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:53.921058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:53.921194 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:53.921583 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:53.921587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:53.921859 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:53.922012 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:53.922023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.923198 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.923202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:53.924208 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:53.924220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:53.924669 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:53.924682 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:53.924795 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:53.925308 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:53.925321 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:53.925631 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:53.927123 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:53.927126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:53.927128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:53.927317 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:53.927545 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:53.927548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:53.927550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:53.927634 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:53.928019 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:53.928023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:53.928117 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 00:44:53.928232 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:44:53.928371 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:44:53.942479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44681
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:54.052112 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:54.052371 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:54.052414 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:54.056708 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:54.056766 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:54.056805 8134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:54.056838 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:54.062016 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:54.062942 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:54.063898 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:54.073829 8142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:54.073841 8139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:54.073829 8140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:54.074271 8134 server_base.cc:1034] running on GCE node
I20260927 00:44:54.074702 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:54.075354 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:54.076551 8134 hybrid_clock.cc:648] HybridClock initialized: now 1790469894076527 us; error 36 us; skew 500 ppm
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:54.079527 8134 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:44:54.080475 8134 webserver.cc:460] Webserver started at http://127.0.98.2:37407/ using document root <none> and password file <none>
I20260927 00:44:54.081048 8134 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:54.081141 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:54.081374 8134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:54.083536 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b225aa8208d41c494c110db83d270d5"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "d17a3e2e434623706a77964c4bc33873"
server_key_iv: "4b1cd77cd52785ff2732ff0e1dfb253a"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.084133 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b225aa8208d41c494c110db83d270d5"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "d17a3e2e434623706a77964c4bc33873"
server_key_iv: "4b1cd77cd52785ff2732ff0e1dfb253a"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.087916 8134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:44:54.090332 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:54.091619 8134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:54.091759 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "8b225aa8208d41c494c110db83d270d5"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "d17a3e2e434623706a77964c4bc33873"
server_key_iv: "4b1cd77cd52785ff2732ff0e1dfb253a"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.091867 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:54.103945 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:54.107136 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:54.107347 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:54.108853 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:54.108928 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:54.109000 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:54.109042 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:54.120118 8134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44887
I20260927 00:44:54.120185 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44887 every 8 connection(s)
I20260927 00:44:54.121012 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:54.123772 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.109720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60191 (local address 127.0.98.62:44681)
0927 00:44:54.109874 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.109878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.110982 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 00:44:54.112126 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.112130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.112208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.112307 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.114087 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.114991 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.116471 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.116731 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.116858 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.120058 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.120064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.120068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.120100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.122019 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.122505 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.122508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.122509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.122585 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.122869 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.122871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.122872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.123037 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.123108 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.123561 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.123656 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:44:54.124650 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.110290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.2:60191)
0927 00:44:54.110828 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.110872 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.111863 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.112573 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.112585 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.113117 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:54.113936 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.113953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.115123 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.115130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.116333 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.116349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.116908 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.116912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.117103 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.117896 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:54.117921 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:54.119869 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:54.122130 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.122134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.122145 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.122406 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.122649 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.122651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.122652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.122788 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.123124 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:54.123129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:54.123354 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.123504 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.123763 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":144,"threads_started":1}
I20260927 00:44:54.125913 8265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44681
I20260927 00:44:54.126086 8265 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:54.126600 8265 heartbeater.cc:511] Master 127.0.98.62:44681 requested a full tablet report, sending...
I20260927 00:44:54.127635 7922 ts_manager.cc:194] Registered new tserver with Master: 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:54.128512 7922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60191
I20260927 00:44:54.129354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8134
I20260927 00:44:54.129496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:54.129899 392 external_mini_cluster.cc:1428] Setting key fb501404696c095a405dbc6661e91259
I20260927 00:44:54.135761 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.127968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.2:46613)
0927 00:44:54.128287 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.128303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.128400 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.128628 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.128631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.128858 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:54.128983 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.128991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.129823 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.129827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.130726 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.130737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.131275 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.131279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.131375 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.131869 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:54.131879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:54.132181 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:54.134153 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.134156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.134158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.134369 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.134665 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.134667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.134668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.134752 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.135241 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:54.135243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:54.135319 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.135402 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.135505 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260927 00:44:54.135870 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.128086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46613 (local address 127.0.98.62:44681)
0927 00:44:54.128217 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.128221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.128376 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:44:54.128483 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.128485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.128519 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.128581 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.129089 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.129673 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.130850 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.131146 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.131275 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.132308 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.132312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.132314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.132336 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.134034 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.134471 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.134477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.134482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.134570 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.134844 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.134850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.134855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.135145 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.135251 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.135500 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.135626 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:44:54.157591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44681
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:54.267081 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:54.267334 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:54.267417 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:54.271850 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:54.271912 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:54.271983 8274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:44:54.272071 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:54.277311 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:54.278370 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:54.279422 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:54.289033 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:54.289059 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:54.289052 8274 server_base.cc:1034] running on GCE node
W20260927 00:44:54.289070 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:54.289656 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:54.290302 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:54.291467 8274 hybrid_clock.cc:648] HybridClock initialized: now 1790469894291451 us; error 26 us; skew 500 ppm
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:54.294030 8274 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:44:54.294920 8274 webserver.cc:460] Webserver started at http://127.0.98.3:39929/ using document root <none> and password file <none>
I20260927 00:44:54.295442 8274 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:54.295498 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:54.295712 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:54.297742 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "82000e1067779c27da69930ad5e4c6d0"
server_key_iv: "c520a12131f2042d6d85251ac45e6499"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.298342 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "82000e1067779c27da69930ad5e4c6d0"
server_key_iv: "c520a12131f2042d6d85251ac45e6499"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.301870 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:54.304215 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:54.305387 8274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:54.305476 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
format_stamp: "Formatted at 2026-09-27 00:44:54 on dist-test-slave-l5sb"
server_key: "82000e1067779c27da69930ad5e4c6d0"
server_key_iv: "c520a12131f2042d6d85251ac45e6499"
server_key_version: "encryptionkey@0"
I20260927 00:44:54.305554 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:54.333246 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:54.336258 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:54.336418 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:54.337914 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:54.337988 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:54.338050 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:54.338109 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:54.349232 8274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37753
I20260927 00:44:54.349277 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37753 every 8 connection(s)
I20260927 00:44:54.350337 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:54.351451 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.338636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43321 (local address 127.0.98.62:44681)
0927 00:44:54.338780 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.338783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.339857 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 00:44:54.340918 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.340922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.340986 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.341055 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.342461 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.343233 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.344598 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.344883 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.344999 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.347581 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.347585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.347587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.347609 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.349300 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.349785 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.349789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.349791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.349886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.350275 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.350281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.350283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.350505 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.350606 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.351167 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.351286 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:44:54.352461 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.339208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.3:43321)
0927 00:44:54.339705 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.339749 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.340650 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.341390 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.341396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.341720 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:54.342244 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.342258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.343364 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.343372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.344474 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.344484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.345046 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.345050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.345220 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.345875 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:54.345906 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:54.347411 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:54.349400 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.349408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.349422 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.349665 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.350023 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.350026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.350027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.350170 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.350615 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:54.350620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:54.350897 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.351089 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.351400 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260927 00:44:54.353467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8274
I20260927 00:44:54.353617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:54.353627 8405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44681
I20260927 00:44:54.353792 8405 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:54.353991 392 external_mini_cluster.cc:1428] Setting key a82a243a4d5db60df043b920ffceecfa
I20260927 00:44:54.354390 8405 heartbeater.cc:511] Master 127.0.98.62:44681 requested a full tablet report, sending...
I20260927 00:44:54.355531 7921 ts_manager.cc:194] Registered new tserver with Master: ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753)
I20260927 00:44:54.356321 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43321
I20260927 00:44:54.362473 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.356009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60419 (local address 127.0.98.62:44681)
0927 00:44:54.356103 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.356107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.356296 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:44:54.356399 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.356402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.356439 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.356520 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.357012 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.357509 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.358406 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.358621 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.358730 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.359722 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.359725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.359728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.359757 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.360875 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.361275 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.361279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.361281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.361357 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.361629 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.361634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.361636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.361864 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.361965 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.362234 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.362337 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:44:54.362485 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.355951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.98.3:60419)
0927 00:44:54.356207 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.356228 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.356303 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.356543 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.356545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.356774 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:54.356894 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.356903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.357631 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.357634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.358288 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.358296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.358733 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.358736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.358833 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.359281 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:54.359291 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:54.359577 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:54.360983 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.360987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.360989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.361180 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.361455 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.361458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.361460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.361543 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.361956 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:54.361959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:54.362067 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.362178 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.362311 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260927 00:44:54.367398 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:44:54.390023 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:54.384007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.98.62:44681)
0927 00:44:54.384120 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.384124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.384134 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:44:54.384176 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.384179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.384215 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.384305 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.385013 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.385531 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.386454 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.386670 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.386782 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.389887 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42482: BlockingRecv error: recv got EOF from 127.0.0.1:42482 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:54.398970 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.391304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.62:44681)
0927 00:44:54.391427 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.391433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.391545 (+ 112us) server_negotiation.cc:407] Connection header received
0927 00:44:54.391632 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.391634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.391672 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.391749 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.392285 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.392767 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.393476 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.393691 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.393761 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.395964 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.395967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.395969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.395991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.397457 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.397865 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.397868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.397869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.397940 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.398165 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.398167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.398168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.398330 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.398444 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.398666 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.398753 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:44:54.401600 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:54.404572 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:54.419174 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.413524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.98.1:44447)
0927 00:44:54.414015 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.414029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.414052 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:54.414124 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.414130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.414318 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.414481 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.414745 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.415919 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.417728 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418551 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.418647 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.418784 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.418917 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.419011 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":69,"threads_started":1}
I20260927 00:44:54.419222 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.412896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44447 (local address 127.0.0.1:59174)
0927 00:44:54.413524 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.413537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.413614 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.414472 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.414477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.414498 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.414616 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.414622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.416020 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.416023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.417597 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.417607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418644 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.418648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.418759 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.418788 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.418880 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.419059 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":191,"thread_start_us":88,"threads_started":1}
I20260927 00:44:54.419557 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.414041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.98.2:44887)
0927 00:44:54.414747 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.414763 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.414779 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:44:54.414834 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.414837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.414955 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.415092 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.415321 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.416077 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.417864 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418498 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.418623 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.418791 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.419282 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.419367 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":128,"threads_started":1}
I20260927 00:44:54.419564 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.412844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.0.98.3:37753)
0927 00:44:54.413683 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.413699 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.413722 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:44:54.413820 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.413825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.414039 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.414528 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.414659 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.415862 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.417554 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418412 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.419117 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.419273 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.419332 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.419430 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":150,"threads_started":1}
I20260927 00:44:54.419811 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.412738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.0.1:33996)
0927 00:44:54.413240 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.413277 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.413376 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.414210 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.414216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.414239 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.414396 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.414406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.415983 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.415986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.417395 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.417404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418567 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.418571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.418701 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.418772 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.418874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.419019 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":194,"thread_start_us":78,"threads_started":1}
I20260927 00:44:54.419222 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.413582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44887 (local address 127.0.0.1:34194)
0927 00:44:54.414427 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.414442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.414509 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.415084 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.415087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.415103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.415209 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.415217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.416192 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.416195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.417733 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.417740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.418852 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.418858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.418952 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.418974 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.419036 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.419104 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":117,"thread_start_us":89,"threads_started":1}
I20260927 00:44:54.422550 8339 tablet_service.cc:1467] Processing CreateTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 (DEFAULT_TABLE table=test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.422546 8059 tablet_service.cc:1467] Processing CreateTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 (DEFAULT_TABLE table=test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.423669 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c8ccb191e24a8d836c5ffd643a4811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.423684 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c8ccb191e24a8d836c5ffd643a4811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.428119 8199 tablet_service.cc:1467] Processing CreateTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 (DEFAULT_TABLE table=test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.429144 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c8ccb191e24a8d836c5ffd643a4811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.431062 8433 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Bootstrap starting.
I20260927 00:44:54.432358 8434 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Bootstrap starting.
I20260927 00:44:54.433657 8433 tablet_bootstrap.cc:654] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.434134 8434 tablet_bootstrap.cc:654] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.434835 8434 log.cc:824] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:54.434916 8433 log.cc:824] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:54.436149 8437 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Bootstrap starting.
I20260927 00:44:54.436894 8434 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: No bootstrap required, opened a new log
I20260927 00:44:54.437085 8434 ts_tablet_manager.cc:1397] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:44:54.438019 8437 tablet_bootstrap.cc:654] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.438656 8437 log.cc:824] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:54.440021 8434 raft_consensus.cc:348] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.440218 8434 raft_consensus.cc:374] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.440284 8437 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: No bootstrap required, opened a new log
I20260927 00:44:54.440286 8434 raft_consensus.cc:729] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.440481 8437 ts_tablet_manager.cc:1397] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 00:44:54.440770 8434 consensus_queue.cc:260] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.441566 8433 tablet_bootstrap.cc:492] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: No bootstrap required, opened a new log
I20260927 00:44:54.441716 8434 ts_tablet_manager.cc:1428] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:44:54.441751 8433 ts_tablet_manager.cc:1397] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260927 00:44:54.442090 8405 heartbeater.cc:503] Master 127.0.98.62:44681 was elected leader, sending a full tablet report...
I20260927 00:44:54.444069 8437 raft_consensus.cc:348] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.444329 8437 raft_consensus.cc:374] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.444379 8437 raft_consensus.cc:729] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.444819 8437 consensus_queue.cc:260] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.445468 8265 heartbeater.cc:503] Master 127.0.98.62:44681 was elected leader, sending a full tablet report...
I20260927 00:44:54.445848 8437 ts_tablet_manager.cc:1428] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 00:44:54.445183 8433 raft_consensus.cc:348] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.446384 8433 raft_consensus.cc:374] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.446444 8433 raft_consensus.cc:729] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29716c38851c4adaa4842959042bd564, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.446831 8433 consensus_queue.cc:260] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.447404 8125 heartbeater.cc:503] Master 127.0.98.62:44681 was elected leader, sending a full tablet report...
I20260927 00:44:54.447746 8433 ts_tablet_manager.cc:1428] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:54.490399 8441 raft_consensus.cc:482] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:54.490593 8441 raft_consensus.cc:504] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.491811 8441 leader_election.cc:288] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:54.497169 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.492214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41805 (local address 127.0.98.3:37753)
0927 00:44:54.492367 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.492372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.492703 (+ 331us) server_negotiation.cc:407] Connection header received
0927 00:44:54.492848 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.492851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.492905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.492989 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.493366 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.494191 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.496172 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.496653 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.496760 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.496814 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.496972 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.497044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:54.497164 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.492077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.1:41805)
0927 00:44:54.492597 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.492612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.492738 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.493015 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.493018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.493054 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.493258 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.493263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.494323 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.494327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.496016 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.496028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.496767 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.496769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.496843 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.496864 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.496952 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.497030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":408,"spinlock_wait_cycles":30464,"thread_start_us":92,"threads_started":1}
I20260927 00:44:54.498034 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.492077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44887 (local address 127.0.98.1:51587)
0927 00:44:54.492564 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.492579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.492731 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.493015 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.493018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.493056 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.493243 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.493251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.494323 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.494327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.496023 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.496028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.497623 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.497627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.497741 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.497760 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.497879 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.497929 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":219,"spinlock_wait_cycles":6272,"thread_start_us":181,"threads_started":1}
I20260927 00:44:54.498081 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.492219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51587 (local address 127.0.98.2:44887)
0927 00:44:54.492372 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.492376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.492706 (+ 330us) server_negotiation.cc:407] Connection header received
0927 00:44:54.492848 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.492851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.492904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.492989 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.493366 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.494191 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.496865 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.497515 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.497604 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.497648 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.497887 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.497957 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:44:54.498013 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c8ccb191e24a8d836c5ffd643a4811" candidate_uuid: "29716c38851c4adaa4842959042bd564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" is_pre_election: true
I20260927 00:44:54.498343 8359 raft_consensus.cc:2454] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29716c38851c4adaa4842959042bd564 in term 0.
I20260927 00:44:54.498780 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c8ccb191e24a8d836c5ffd643a4811" candidate_uuid: "29716c38851c4adaa4842959042bd564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5" is_pre_election: true
I20260927 00:44:54.498826 8011 leader_election.cc:302] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:54.499059 8219 raft_consensus.cc:2454] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29716c38851c4adaa4842959042bd564 in term 0.
I20260927 00:44:54.499107 8441 raft_consensus.cc:2781] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:54.499197 8441 raft_consensus.cc:482] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:54.499253 8441 raft_consensus.cc:3037] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.500293 8441 raft_consensus.cc:504] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.500636 8441 leader_election.cc:288] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [CANDIDATE]: Term 1 election: Requested vote from peers ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:54.500964 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c8ccb191e24a8d836c5ffd643a4811" candidate_uuid: "29716c38851c4adaa4842959042bd564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5"
I20260927 00:44:54.501127 8219 raft_consensus.cc:3037] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.501133 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c8ccb191e24a8d836c5ffd643a4811" candidate_uuid: "29716c38851c4adaa4842959042bd564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
I20260927 00:44:54.501260 8359 raft_consensus.cc:3037] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.502336 8359 raft_consensus.cc:2454] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29716c38851c4adaa4842959042bd564 in term 1.
I20260927 00:44:54.502592 8219 raft_consensus.cc:2454] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29716c38851c4adaa4842959042bd564 in term 1.
I20260927 00:44:54.502617 8011 leader_election.cc:302] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:54.502811 8441 raft_consensus.cc:2781] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:54.503023 8441 raft_consensus.cc:686] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 1 LEADER]: Becoming Leader. State: Replica: 29716c38851c4adaa4842959042bd564, State: Running, Role: LEADER
I20260927 00:44:54.503350 8441 consensus_queue.cc:237] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } }
I20260927 00:44:54.506515 7921 catalog_manager.cc:5495] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29716c38851c4adaa4842959042bd564 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29716c38851c4adaa4842959042bd564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:54.539507 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.535872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.0.98.1:44447)
0927 00:44:54.536014 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.536019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.536034 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:54.536208 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.536211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.536270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.536387 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.536628 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.537131 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.538021 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.538250 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.538362 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.538644 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.539095 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.539230 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.539297 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.539354 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:44:54.545624 8219 raft_consensus.cc:1261] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Refusing update from remote peer 29716c38851c4adaa4842959042bd564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.545951 8359 raft_consensus.cc:1261] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Refusing update from remote peer 29716c38851c4adaa4842959042bd564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.546355 8444 consensus_queue.cc:1035] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.546550 8441 consensus_queue.cc:1035] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.555511 8449 mvcc.cc:205] Tried to move back new op lower bound from 7333764688051761152 to 7333764687888375808. Current Snapshot: MvccSnapshot[applied={T|T < 7333764688051761152}]
I20260927 00:44:54.556711 8448 mvcc.cc:205] Tried to move back new op lower bound from 7333764688051761152 to 7333764687888375808. Current Snapshot: MvccSnapshot[applied={T|T < 7333764688051761152}]
I20260927 00:44:54.563266 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.563422 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.565804 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Sending DeleteTablet for 3 replicas of tablet b7c8ccb191e24a8d836c5ffd643a4811
I20260927 00:44:54.566599 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260927 00:44:54.566648 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33996
I20260927 00:44:54.567317 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c8ccb191e24a8d836c5ffd643a4811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 00:44:54.567483 8456 tablet_replica.cc:330] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: stopping tablet replica
I20260927 00:44:54.567751 8458 tablet_replica.cc:330] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: stopping tablet replica
I20260927 00:44:54.567864 8456 raft_consensus.cc:2229] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.567988 8458 raft_consensus.cc:2229] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:54.568234 8456 raft_consensus.cc:2258] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.568359 8458 raft_consensus.cc:2258] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.568272 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:54.568718 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:54.568787 8457 tablet_replica.cc:330] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: stopping tablet replica
I20260927 00:44:54.569051 8457 raft_consensus.cc:2229] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.569332 8457 raft_consensus.cc:2258] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.570524 8456 ts_tablet_manager.cc:1905] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.570719 8457 ts_tablet_manager.cc:1905] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.573305 8457 ts_tablet_manager.cc:1918] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.573307 8456 ts_tablet_manager.cc:1918] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.573401 8457 log.cc:1196] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/b7c8ccb191e24a8d836c5ffd643a4811
I20260927 00:44:54.573405 8456 log.cc:1196] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/b7c8ccb191e24a8d836c5ffd643a4811
I20260927 00:44:54.573714 8456 ts_tablet_manager.cc:1939] T b7c8ccb191e24a8d836c5ffd643a4811 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting consensus metadata
I20260927 00:44:54.574286 8199 tablet_service.cc:1467] Processing CreateTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 (DEFAULT_TABLE table=test-table [id=d4726fff509145f0b4a4471545e6fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.574294 8339 tablet_service.cc:1467] Processing CreateTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 (DEFAULT_TABLE table=test-table [id=d4726fff509145f0b4a4471545e6fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.574525 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c3d9e48a394629a1ae2d5cb431ed81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.574527 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c3d9e48a394629a1ae2d5cb431ed81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.576269 8437 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Bootstrap starting.
I20260927 00:44:54.576484 8434 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Bootstrap starting.
I20260927 00:44:54.576473 8059 tablet_service.cc:1467] Processing CreateTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 (DEFAULT_TABLE table=test-table [id=d4726fff509145f0b4a4471545e6fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.576745 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c3d9e48a394629a1ae2d5cb431ed81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.576885 7907 catalog_manager.cc:4841] TS ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753): tablet b7c8ccb191e24a8d836c5ffd643a4811 (table test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]) successfully deleted
I20260927 00:44:54.577235 8437 tablet_bootstrap.cc:654] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.577433 8434 tablet_bootstrap.cc:654] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.578773 8458 ts_tablet_manager.cc:1905] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.579141 8434 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: No bootstrap required, opened a new log
I20260927 00:44:54.579221 8434 ts_tablet_manager.cc:1397] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:54.579582 8457 ts_tablet_manager.cc:1939] T b7c8ccb191e24a8d836c5ffd643a4811 P 8b225aa8208d41c494c110db83d270d5: Deleting consensus metadata
I20260927 00:44:54.579628 8434 raft_consensus.cc:348] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.579766 8434 raft_consensus.cc:374] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.579799 8434 raft_consensus.cc:729] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.579916 8434 consensus_queue.cc:260] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.580147 8434 ts_tablet_manager.cc:1428] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.580374 8433 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Bootstrap starting.
I20260927 00:44:54.580830 7909 catalog_manager.cc:4841] TS 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887): tablet b7c8ccb191e24a8d836c5ffd643a4811 (table test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]) successfully deleted
I20260927 00:44:54.581234 8458 ts_tablet_manager.cc:1918] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.581332 8458 log.cc:1196] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/b7c8ccb191e24a8d836c5ffd643a4811
I20260927 00:44:54.581645 8458 ts_tablet_manager.cc:1939] T b7c8ccb191e24a8d836c5ffd643a4811 P 29716c38851c4adaa4842959042bd564: Deleting consensus metadata
I20260927 00:44:54.581689 8437 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: No bootstrap required, opened a new log
I20260927 00:44:54.581763 8437 ts_tablet_manager.cc:1397] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:44:54.582195 8437 raft_consensus.cc:348] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.582297 8437 raft_consensus.cc:374] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.582329 8437 raft_consensus.cc:729] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.582453 8437 consensus_queue.cc:260] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.582720 8437 ts_tablet_manager.cc:1428] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.582753 7906 catalog_manager.cc:4841] TS 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447): tablet b7c8ccb191e24a8d836c5ffd643a4811 (table test-table [id=ecc671e0db3d4ad6b71aa8ff8dbd8953]) successfully deleted
I20260927 00:44:54.581245 8433 tablet_bootstrap.cc:654] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.584753 8433 tablet_bootstrap.cc:492] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: No bootstrap required, opened a new log
I20260927 00:44:54.584851 8433 ts_tablet_manager.cc:1397] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 00:44:54.585301 8433 raft_consensus.cc:348] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.585408 8433 raft_consensus.cc:374] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.585453 8433 raft_consensus.cc:729] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29716c38851c4adaa4842959042bd564, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.585588 8433 consensus_queue.cc:260] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.585872 8433 ts_tablet_manager.cc:1428] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 00:44:54.601307 8406 tablet.cc:2369] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:54.612413 8439 raft_consensus.cc:482] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:54.612546 8439 raft_consensus.cc:504] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.613732 8439 leader_election.cc:288] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:54.618077 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.614163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34663 (local address 127.0.98.1:44447)
0927 00:44:54.614291 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.614296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.614529 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:44:54.614639 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.614642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.614689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.614759 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.615110 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.615836 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.616874 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.617384 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.617486 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.617548 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.617871 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.617950 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:44:54.618299 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.614364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43437 (local address 127.0.98.2:44887)
0927 00:44:54.614516 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.614520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.614533 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:54.614585 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.614589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.614641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.614739 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.614986 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.615772 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.617208 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.617762 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.617887 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.617942 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.618119 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.618196 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:44:54.618299 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.613949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44887 (local address 127.0.98.3:43437)
0927 00:44:54.614080 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.614097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.614193 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.614750 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.614753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.614767 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.614886 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.614892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.615885 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.615888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.617075 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.617085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.617887 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.617891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.617987 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.618007 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.618103 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.618176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:44:54.618734 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c3d9e48a394629a1ae2d5cb431ed81" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5" is_pre_election: true
I20260927 00:44:54.618912 8219 raft_consensus.cc:2454] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 0.
I20260927 00:44:54.618994 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.613949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44447 (local address 127.0.98.3:34663)
0927 00:44:54.614426 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.614441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.614542 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.614860 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.614863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.614880 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.614998 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.615004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.615959 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.615962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.616749 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.616757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.617518 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.617522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.617620 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.617645 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.618734 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.618808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":96,"thread_start_us":64,"threads_started":1}
I20260927 00:44:54.619290 8293 leader_election.cc:302] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b225aa8208d41c494c110db83d270d5, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:54.619541 8439 raft_consensus.cc:2781] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:54.619495 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c3d9e48a394629a1ae2d5cb431ed81" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564" is_pre_election: true
I20260927 00:44:54.619606 8439 raft_consensus.cc:482] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:54.619643 8439 raft_consensus.cc:3037] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.619729 8079 raft_consensus.cc:2454] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 0.
I20260927 00:44:54.620864 8439 raft_consensus.cc:504] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.621259 8439 leader_election.cc:288] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 election: Requested vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:54.621623 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c3d9e48a394629a1ae2d5cb431ed81" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564"
I20260927 00:44:54.621609 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c3d9e48a394629a1ae2d5cb431ed81" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5"
I20260927 00:44:54.621727 8079 raft_consensus.cc:3037] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.621727 8219 raft_consensus.cc:3037] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:44:54.622331 8266 tablet.cc:2369] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:54.622851 8219 raft_consensus.cc:2454] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 1.
I20260927 00:44:54.622851 8079 raft_consensus.cc:2454] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 1.
I20260927 00:44:54.623194 8290 leader_election.cc:302] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:54.623353 8439 raft_consensus.cc:2781] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:54.623590 8439 raft_consensus.cc:686] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 LEADER]: Becoming Leader. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Running, Role: LEADER
I20260927 00:44:54.623839 8439 consensus_queue.cc:237] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.626039 7921 catalog_manager.cc:5495] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: HEALTHY } } }
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:54.664099 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.655782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44681 (local address 127.0.0.1:42506)
0927 00:44:54.656089 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.656105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.656230 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.656484 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.656487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.656909 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:54.657007 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.657014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.657758 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.657764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.658359 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.658367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.658805 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.658808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.658894 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.659493 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:54.659505 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:54.660877 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:54.662542 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.662552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.662558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.662782 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.663053 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:54.663058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:54.663060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:54.663141 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:54.663511 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:54.663518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:54.663626 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.663744 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.663903 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
I20260927 00:44:54.664098 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.655831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.98.62:44681)
0927 00:44:54.655981 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.655987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.656183 (+ 196us) server_negotiation.cc:407] Connection header received
0927 00:44:54.656336 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.656340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.656385 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.656469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.657132 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.657645 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.658478 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.658693 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.658801 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.661016 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.661019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.661021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.661040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.662437 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.662886 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.662890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.662893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.662964 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.663230 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.663234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.663236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.663406 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.663512 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.663797 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.663955 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":46}
W20260927 00:44:54.665330 8126 tablet.cc:2369] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:54.668502 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42506:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:44:54.669282 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:54.674185 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 170281f0a9c1483caed3260a4cf9f8ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf0efecd38b44e3ab33348c4f8f479fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:54.674413 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 170281f0a9c1483caed3260a4cf9f8ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf0efecd38b44e3ab33348c4f8f479fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:54.674592 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170281f0a9c1483caed3260a4cf9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.674760 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170281f0a9c1483caed3260a4cf9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.676898 8433 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: Bootstrap starting.
I20260927 00:44:54.677712 8434 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: Bootstrap starting.
I20260927 00:44:54.677860 8433 tablet_bootstrap.cc:654] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.678119 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 170281f0a9c1483caed3260a4cf9f8ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf0efecd38b44e3ab33348c4f8f479fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:44:54.678481 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170281f0a9c1483caed3260a4cf9f8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.678725 8434 tablet_bootstrap.cc:654] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.680226 8434 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: No bootstrap required, opened a new log
I20260927 00:44:54.680328 8434 ts_tablet_manager.cc:1397] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:54.680723 8434 raft_consensus.cc:348] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.680850 8434 raft_consensus.cc:374] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.680891 8434 raft_consensus.cc:729] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.681031 8434 consensus_queue.cc:260] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.681353 8437 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: Bootstrap starting.
I20260927 00:44:54.681422 8434 ts_tablet_manager.cc:1428] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.682277 8437 tablet_bootstrap.cc:654] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.682374 8433 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: No bootstrap required, opened a new log
I20260927 00:44:54.682451 8433 ts_tablet_manager.cc:1397] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:44:54.682870 8433 raft_consensus.cc:348] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.682891 8439 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.682977 8433 raft_consensus.cc:374] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.683007 8433 raft_consensus.cc:729] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29716c38851c4adaa4842959042bd564, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.683012 8439 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.683117 8433 consensus_queue.cc:260] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.683415 8433 ts_tablet_manager.cc:1428] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.683619 8441 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.683735 8441 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.685566 8437 tablet_bootstrap.cc:492] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: No bootstrap required, opened a new log
I20260927 00:44:54.685683 8437 ts_tablet_manager.cc:1397] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 00:44:54.686113 8437 raft_consensus.cc:348] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.686244 8437 raft_consensus.cc:374] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.686290 8437 raft_consensus.cc:729] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.686427 8437 consensus_queue.cc:260] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.686702 8437 ts_tablet_manager.cc:1428] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:44:54.686642 8440 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.686784 8440 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.739970 8440 raft_consensus.cc:482] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:54.740124 8440 raft_consensus.cc:504] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.741240 8440 leader_election.cc:288] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753)
I20260927 00:44:54.745206 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.741625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60699 (local address 127.0.98.3:37753)
0927 00:44:54.741793 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.741799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.741920 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:44:54.742104 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.742107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.742164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.742247 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.742512 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.743012 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744147 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.744546 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744668 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.744737 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.744983 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.745061 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:44:54.745205 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.741537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45909 (local address 127.0.98.1:44447)
0927 00:44:54.741695 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.741700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.741916 (+ 216us) server_negotiation.cc:407] Connection header received
0927 00:44:54.742102 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.742105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.742155 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.742242 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:54.742540 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.743033 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744162 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.744557 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744680 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.744750 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.744984 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.745060 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:44:54.745256 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.741482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.2:60699)
0927 00:44:54.741822 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.741837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.741992 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.742260 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.742262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.742278 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.742406 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.742412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.743129 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.743133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.744020 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.744029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.744680 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.744827 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.744883 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.744969 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.745068 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":62,"spinlock_wait_cycles":11648,"thread_start_us":86,"threads_started":1}
I20260927 00:44:54.745265 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.741454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44447 (local address 127.0.98.2:45909)
0927 00:44:54.741831 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.741843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.741992 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.742253 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.742255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.742270 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:54.742423 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.742430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.743144 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.743149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.744034 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.744042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.744687 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.744690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.744854 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.744880 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.744967 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.745060 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":118,"spinlock_wait_cycles":59904,"thread_start_us":89,"threads_started":1}
I20260927 00:44:54.745694 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170281f0a9c1483caed3260a4cf9f8ce" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564" is_pre_election: true
I20260927 00:44:54.745695 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170281f0a9c1483caed3260a4cf9f8ce" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" is_pre_election: true
I20260927 00:44:54.745867 8079 raft_consensus.cc:2454] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 0.
I20260927 00:44:54.745869 8359 raft_consensus.cc:2454] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 0.
I20260927 00:44:54.746239 8150 leader_election.cc:302] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, 8b225aa8208d41c494c110db83d270d5; no voters:
I20260927 00:44:54.746531 8440 raft_consensus.cc:2781] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:54.746609 8440 raft_consensus.cc:482] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:54.746665 8440 raft_consensus.cc:3037] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.747650 8440 raft_consensus.cc:504] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.748016 8440 leader_election.cc:288] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 election: Requested vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753)
I20260927 00:44:54.748412 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170281f0a9c1483caed3260a4cf9f8ce" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564"
I20260927 00:44:54.748423 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170281f0a9c1483caed3260a4cf9f8ce" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
I20260927 00:44:54.748543 8079 raft_consensus.cc:3037] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.748554 8359 raft_consensus.cc:3037] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.749430 8079 raft_consensus.cc:2454] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 1.
I20260927 00:44:54.749457 8359 raft_consensus.cc:2454] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 1.
I20260927 00:44:54.749723 8150 leader_election.cc:302] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, 8b225aa8208d41c494c110db83d270d5; no voters:
I20260927 00:44:54.749927 8440 raft_consensus.cc:2781] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:54.750123 8440 raft_consensus.cc:686] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Running, Role: LEADER
I20260927 00:44:54.750321 8440 consensus_queue.cc:237] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.751585 8476 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: TxnStatusTablet state changed. Reason: New leader 8b225aa8208d41c494c110db83d270d5. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.751711 8476 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:54.751978 8478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:44:54.752136 8478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:44:54.752812 7921 catalog_manager.cc:5495] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b225aa8208d41c494c110db83d270d5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:54.764746 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.760762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.0.98.2:44887)
0927 00:44:54.760879 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.760882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.760973 (+ 91us) server_negotiation.cc:407] Connection header received
0927 00:44:54.761152 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.761154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.761206 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.761287 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.761621 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.762220 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.763426 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.763632 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.763705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.764014 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.764378 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.764459 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.764585 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.764642 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:44:54.764796 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.760660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44887 (local address 127.0.0.1:34196)
0927 00:44:54.760896 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.760911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.761027 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.761330 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.761333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.761344 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:54.761455 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.761462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.762335 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.762338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.763315 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.763326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.763736 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.763740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.763834 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.763888 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.764475 (+ 587us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.764490 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.764571 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.764657 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 00:44:54.767582 8359 raft_consensus.cc:1261] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Refusing update from remote peer 8b225aa8208d41c494c110db83d270d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.767725 8079 raft_consensus.cc:1261] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Refusing update from remote peer 8b225aa8208d41c494c110db83d270d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.768154 8440 consensus_queue.cc:1035] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.768342 8476 consensus_queue.cc:1035] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.769691 8439 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: TxnStatusTablet state changed. Reason: New leader 8b225aa8208d41c494c110db83d270d5. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.770102 8439 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.775592 8441 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: TxnStatusTablet state changed. Reason: New leader 8b225aa8208d41c494c110db83d270d5. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.775722 8441 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.777393 8476 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.777565 8476 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:54.778851 8447 mvcc.cc:205] Tried to move back new op lower bound from 7333764688963207168 to 7333764688899985408. Current Snapshot: MvccSnapshot[applied={T|T < 7333764688963207168 or (T in {7333764688963207168})}]
I20260927 00:44:54.779078 8441 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.779197 8441 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 29716c38851c4adaa4842959042bd564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.781433 8476 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.781534 8476 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P 8b225aa8208d41c494c110db83d270d5: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:44:54.782021 8439 tablet_replica.cc:439] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } } }
I20260927 00:44:54.782126 8439 tablet_replica.cc:442] T 170281f0a9c1483caed3260a4cf9f8ce P ea9ad3d6b21841fcb77153d1df462b0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:44:54.787348 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.782783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.0.98.3:37753)
0927 00:44:54.782890 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.782895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.782970 (+ 75us) server_negotiation.cc:407] Connection header received
0927 00:44:54.783178 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.783182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.783247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.783340 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.784343 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.784856 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.786007 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.786227 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.786307 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.786589 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.786998 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.787081 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.787168 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.787211 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:44:54.797936 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.794278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35917 (local address 127.0.98.2:44887)
0927 00:44:54.794394 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.794398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.794439 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:44:54.794574 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.794576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.794631 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.794692 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.794957 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.795621 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.796848 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.797124 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.797206 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.797505 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.797568 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.797635 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.797747 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.797836 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:44:54.797968 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.794187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44887 (local address 127.0.98.3:35917)
0927 00:44:54.794383 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.794397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.794484 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.794723 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.794726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.794734 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:54.794856 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.794862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.795718 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.795722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.796733 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.796744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.797227 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.797231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.797329 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.797379 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.797663 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.797670 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.797736 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.797833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 00:44:54.805157 8079 raft_consensus.cc:1261] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Refusing update from remote peer ea9ad3d6b21841fcb77153d1df462b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.805202 8219 raft_consensus.cc:1261] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Refusing update from remote peer ea9ad3d6b21841fcb77153d1df462b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.805612 8439 consensus_queue.cc:1035] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.805825 8463 consensus_queue.cc:1035] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.829792 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.825975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42785 (local address 127.0.98.3:37753)
0927 00:44:54.826075 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.826080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.826155 (+ 75us) server_negotiation.cc:407] Connection header received
0927 00:44:54.826310 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.826315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.826367 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.826465 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.826714 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.827533 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.828631 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.828964 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.829067 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.829352 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.829436 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.829502 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.829586 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.829633 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:44:54.829818 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.825617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37753 (local address 127.0.98.2:42785)
0927 00:44:54.826054 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:54.826070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:54.826204 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:54.826498 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:54.826500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:54.826508 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:44:54.826616 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.826621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.827640 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.827643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.828507 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:54.828516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.829053 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.829059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:54.829151 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.829205 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.829504 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.829512 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:44:54.829576 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:44:54.829651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
I20260927 00:44:54.855942 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.856113 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.858268 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Sending DeleteTablet for 3 replicas of tablet c2c3d9e48a394629a1ae2d5cb431ed81
I20260927 00:44:54.858789 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 00:44:54.859002 8458 tablet_replica.cc:330] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: stopping tablet replica
I20260927 00:44:54.859170 8458 raft_consensus.cc:2229] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.859241 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33996
I20260927 00:44:54.859293 8458 raft_consensus.cc:2258] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.859462 8456 tablet_replica.cc:330] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: stopping tablet replica
I20260927 00:44:54.859557 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c3d9e48a394629a1ae2d5cb431ed81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260927 00:44:54.859589 8456 raft_consensus.cc:2229] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:54.859721 8457 tablet_replica.cc:330] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: stopping tablet replica
I20260927 00:44:54.859831 8457 raft_consensus.cc:2229] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.859876 8456 raft_consensus.cc:2258] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.859947 8457 raft_consensus.cc:2258] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.860661 8456 ts_tablet_manager.cc:1905] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.860747 8457 ts_tablet_manager.cc:1905] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.863535 8456 ts_tablet_manager.cc:1918] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:54.863636 8456 log.cc:1196] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/c2c3d9e48a394629a1ae2d5cb431ed81
I20260927 00:44:54.863917 8456 ts_tablet_manager.cc:1939] T c2c3d9e48a394629a1ae2d5cb431ed81 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting consensus metadata
I20260927 00:44:54.863909 8457 ts_tablet_manager.cc:1918] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:54.863981 8457 log.cc:1196] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/c2c3d9e48a394629a1ae2d5cb431ed81
I20260927 00:44:54.864253 8457 ts_tablet_manager.cc:1939] T c2c3d9e48a394629a1ae2d5cb431ed81 P 8b225aa8208d41c494c110db83d270d5: Deleting consensus metadata
I20260927 00:44:54.864609 7907 catalog_manager.cc:4841] TS ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753): tablet c2c3d9e48a394629a1ae2d5cb431ed81 (table test-table [id=d4726fff509145f0b4a4471545e6fc07]) successfully deleted
I20260927 00:44:54.864931 7909 catalog_manager.cc:4841] TS 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887): tablet c2c3d9e48a394629a1ae2d5cb431ed81 (table test-table [id=d4726fff509145f0b4a4471545e6fc07]) successfully deleted
I20260927 00:44:54.865229 8458 ts_tablet_manager.cc:1905] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.867877 8458 ts_tablet_manager.cc:1918] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:54.867964 8458 log.cc:1196] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/c2c3d9e48a394629a1ae2d5cb431ed81
I20260927 00:44:54.868216 8458 ts_tablet_manager.cc:1939] T c2c3d9e48a394629a1ae2d5cb431ed81 P 29716c38851c4adaa4842959042bd564: Deleting consensus metadata
I20260927 00:44:54.868866 7906 catalog_manager.cc:4841] TS 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447): tablet c2c3d9e48a394629a1ae2d5cb431ed81 (table test-table [id=d4726fff509145f0b4a4471545e6fc07]) successfully deleted
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:44:54.895044 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:54.886702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.0.98.62:44681)
0927 00:44:54.886850 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.886855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.887109 (+ 254us) server_negotiation.cc:407] Connection header received
0927 00:44:54.887251 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.887254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.887313 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.887416 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.888311 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.889197 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.890490 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.890834 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.890964 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.894885 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42512: BlockingRecv error: recv got EOF from 127.0.0.1:42512 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:44:54 dist-test-slave-l5sb krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:44:54.907925 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.897548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.0.98.62:44681)
0927 00:44:54.897682 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.897687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.897703 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:44:54.897822 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.897826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.897876 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.897968 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:54.898735 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.899567 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.900729 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.901085 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.901191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.904136 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:54.904140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:54.904143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:54.904174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:54.906019 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.906513 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.906517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.906519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.906617 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:54.906976 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:54.906979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:54.906981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:54.907218 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:54.907358 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.907614 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.907753 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:44:54.911082 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:54.911571 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:54.917467 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 011a4e3d66414f009ad90de504bb49f3 (DEFAULT_TABLE table=test-table [id=b901bbf772b247bba6322685fb50a235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.917871 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011a4e3d66414f009ad90de504bb49f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.920778 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 011a4e3d66414f009ad90de504bb49f3 (DEFAULT_TABLE table=test-table [id=b901bbf772b247bba6322685fb50a235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.921057 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011a4e3d66414f009ad90de504bb49f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.923470 8437 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Bootstrap starting.
I20260927 00:44:54.924115 8433 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Bootstrap starting.
I20260927 00:44:54.924391 8437 tablet_bootstrap.cc:654] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.924932 8433 tablet_bootstrap.cc:654] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.925292 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 011a4e3d66414f009ad90de504bb49f3 (DEFAULT_TABLE table=test-table [id=b901bbf772b247bba6322685fb50a235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:54.925542 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011a4e3d66414f009ad90de504bb49f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:54.927837 8434 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Bootstrap starting.
I20260927 00:44:54.928632 8434 tablet_bootstrap.cc:654] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:54.930526 8434 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: No bootstrap required, opened a new log
I20260927 00:44:54.930606 8434 ts_tablet_manager.cc:1397] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:44:54.931046 8434 raft_consensus.cc:348] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.931164 8434 raft_consensus.cc:374] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.931198 8434 raft_consensus.cc:729] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.931308 8434 consensus_queue.cc:260] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.931586 8434 ts_tablet_manager.cc:1428] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.938544 8437 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: No bootstrap required, opened a new log
I20260927 00:44:54.938656 8437 ts_tablet_manager.cc:1397] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260927 00:44:54.939088 8437 raft_consensus.cc:348] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.939206 8437 raft_consensus.cc:374] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.939209 8433 tablet_bootstrap.cc:492] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: No bootstrap required, opened a new log
I20260927 00:44:54.939237 8437 raft_consensus.cc:729] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.939291 8433 ts_tablet_manager.cc:1397] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260927 00:44:54.939354 8437 consensus_queue.cc:260] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.939601 8437 ts_tablet_manager.cc:1428] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.939672 8433 raft_consensus.cc:348] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.939786 8433 raft_consensus.cc:374] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:54.939814 8433 raft_consensus.cc:729] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29716c38851c4adaa4842959042bd564, State: Initialized, Role: FOLLOWER
I20260927 00:44:54.939918 8433 consensus_queue.cc:260] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.940176 8433 ts_tablet_manager.cc:1428] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:54.954087 8440 raft_consensus.cc:482] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:54.954226 8440 raft_consensus.cc:504] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.954638 8440 leader_election.cc:288] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753)
I20260927 00:44:54.955083 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011a4e3d66414f009ad90de504bb49f3" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564" is_pre_election: true
I20260927 00:44:54.955081 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011a4e3d66414f009ad90de504bb49f3" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" is_pre_election: true
I20260927 00:44:54.955219 8359 raft_consensus.cc:2454] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 0.
I20260927 00:44:54.955219 8078 raft_consensus.cc:2454] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 0.
I20260927 00:44:54.955571 8150 leader_election.cc:302] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, 8b225aa8208d41c494c110db83d270d5; no voters:
I20260927 00:44:54.955806 8440 raft_consensus.cc:2781] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:54.955870 8440 raft_consensus.cc:482] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:54.955906 8440 raft_consensus.cc:3037] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.957108 8440 raft_consensus.cc:504] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.957456 8440 leader_election.cc:288] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 election: Requested vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753)
I20260927 00:44:54.957805 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011a4e3d66414f009ad90de504bb49f3" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564"
I20260927 00:44:54.957931 8078 raft_consensus.cc:3037] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.957864 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011a4e3d66414f009ad90de504bb49f3" candidate_uuid: "8b225aa8208d41c494c110db83d270d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ad3d6b21841fcb77153d1df462b0e"
I20260927 00:44:54.958019 8359 raft_consensus.cc:3037] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:54.958946 8359 raft_consensus.cc:2454] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 1.
I20260927 00:44:54.959061 8078 raft_consensus.cc:2454] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b225aa8208d41c494c110db83d270d5 in term 1.
I20260927 00:44:54.959321 8151 leader_election.cc:302] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b225aa8208d41c494c110db83d270d5, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:54.959556 8440 raft_consensus.cc:2781] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:54.959683 8440 raft_consensus.cc:686] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Running, Role: LEADER
I20260927 00:44:54.959822 8440 consensus_queue.cc:237] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:54.961447 7921 catalog_manager.cc:5495] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b225aa8208d41c494c110db83d270d5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8b225aa8208d41c494c110db83d270d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:54.975263 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:54.972047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.0.98.2:44887)
0927 00:44:54.972165 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:54.972170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:54.972214 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:44:54.972356 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:54.972359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:54.972399 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:54.972462 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:54.972716 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.973204 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.974206 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:54.974526 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:54.974634 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:54.974856 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:54.974934 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:54.975006 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:54.975080 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:44:54.975133 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:44:54.977384 8359 raft_consensus.cc:1261] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Refusing update from remote peer 8b225aa8208d41c494c110db83d270d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.977510 8078 raft_consensus.cc:1261] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Refusing update from remote peer 8b225aa8208d41c494c110db83d270d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:54.977800 8485 consensus_queue.cc:1035] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.977955 8440 consensus_queue.cc:1035] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:54.987277 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.987450 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:54.989662 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Sending DeleteTablet for 3 replicas of tablet 011a4e3d66414f009ad90de504bb49f3
I20260927 00:44:54.990247 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 011a4e3d66414f009ad90de504bb49f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260927 00:44:54.990427 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 011a4e3d66414f009ad90de504bb49f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 00:44:54.990583 8457 tablet_replica.cc:330] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: stopping tablet replica
I20260927 00:44:54.990631 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 011a4e3d66414f009ad90de504bb49f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33996
I20260927 00:44:54.990705 8457 raft_consensus.cc:2229] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:54.990793 8456 tablet_replica.cc:330] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: stopping tablet replica
I20260927 00:44:54.990901 8456 raft_consensus.cc:2229] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.990967 8457 raft_consensus.cc:2258] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.991024 8456 raft_consensus.cc:2258] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.991129 8458 tablet_replica.cc:330] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: stopping tablet replica
I20260927 00:44:54.991254 8458 raft_consensus.cc:2229] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:54.991374 8458 raft_consensus.cc:2258] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:54.991957 8456 ts_tablet_manager.cc:1905] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.992525 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:54.993069 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:54.994228 8457 ts_tablet_manager.cc:1905] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.994661 8456 ts_tablet_manager.cc:1918] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.994736 8456 log.cc:1196] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/011a4e3d66414f009ad90de504bb49f3
I20260927 00:44:54.994774 8458 ts_tablet_manager.cc:1905] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:54.996043 8456 ts_tablet_manager.cc:1939] T 011a4e3d66414f009ad90de504bb49f3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting consensus metadata
I20260927 00:44:54.996628 8457 ts_tablet_manager.cc:1918] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.996697 8457 log.cc:1196] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/011a4e3d66414f009ad90de504bb49f3
I20260927 00:44:54.996933 8457 ts_tablet_manager.cc:1939] T 011a4e3d66414f009ad90de504bb49f3 P 8b225aa8208d41c494c110db83d270d5: Deleting consensus metadata
I20260927 00:44:54.997488 8458 ts_tablet_manager.cc:1918] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:54.997558 8458 log.cc:1196] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/011a4e3d66414f009ad90de504bb49f3
I20260927 00:44:54.998087 7909 catalog_manager.cc:4841] TS 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887): tablet 011a4e3d66414f009ad90de504bb49f3 (table test-table [id=b901bbf772b247bba6322685fb50a235]) successfully deleted
I20260927 00:44:54.998512 7907 catalog_manager.cc:4841] TS ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753): tablet 011a4e3d66414f009ad90de504bb49f3 (table test-table [id=b901bbf772b247bba6322685fb50a235]) successfully deleted
I20260927 00:44:54.998761 8458 ts_tablet_manager.cc:1939] T 011a4e3d66414f009ad90de504bb49f3 P 29716c38851c4adaa4842959042bd564: Deleting consensus metadata
I20260927 00:44:54.999831 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 (DEFAULT_TABLE table=test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:55.000080 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e3a4f4b32240dea9f7b3081a92ada3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:55.002164 8437 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Bootstrap starting.
I20260927 00:44:55.002343 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 (DEFAULT_TABLE table=test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:55.002556 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e3a4f4b32240dea9f7b3081a92ada3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:55.003007 8437 tablet_bootstrap.cc:654] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:55.004631 8433 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Bootstrap starting.
I20260927 00:44:55.005419 8433 tablet_bootstrap.cc:654] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:55.005568 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 (DEFAULT_TABLE table=test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:55.005827 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e3a4f4b32240dea9f7b3081a92ada3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:55.006502 8433 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: No bootstrap required, opened a new log
I20260927 00:44:55.006592 8433 ts_tablet_manager.cc:1397] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:55.007007 8433 raft_consensus.cc:348] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.007120 8433 raft_consensus.cc:374] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:55.007155 8433 raft_consensus.cc:729] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29716c38851c4adaa4842959042bd564, State: Initialized, Role: FOLLOWER
I20260927 00:44:55.007280 8433 consensus_queue.cc:260] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.007557 8433 ts_tablet_manager.cc:1428] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:55.007999 8434 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Bootstrap starting.
I20260927 00:44:55.008870 8434 tablet_bootstrap.cc:654] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:55.010531 8437 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: No bootstrap required, opened a new log
I20260927 00:44:55.010639 8437 ts_tablet_manager.cc:1397] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260927 00:44:55.010880 8434 tablet_bootstrap.cc:492] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: No bootstrap required, opened a new log
I20260927 00:44:55.010947 8434 ts_tablet_manager.cc:1397] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:44:55.011049 8437 raft_consensus.cc:348] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.011188 8437 raft_consensus.cc:374] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:55.011224 8437 raft_consensus.cc:729] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b225aa8208d41c494c110db83d270d5, State: Initialized, Role: FOLLOWER
I20260927 00:44:55.011348 8437 consensus_queue.cc:260] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.011356 8434 raft_consensus.cc:348] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.011456 8434 raft_consensus.cc:374] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:55.011484 8434 raft_consensus.cc:729] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Initialized, Role: FOLLOWER
I20260927 00:44:55.011588 8434 consensus_queue.cc:260] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.011624 8437 ts_tablet_manager.cc:1428] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:55.011852 8434 ts_tablet_manager.cc:1428] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:44:55.013870 7906 catalog_manager.cc:4841] TS 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447): tablet 011a4e3d66414f009ad90de504bb49f3 (table test-table [id=b901bbf772b247bba6322685fb50a235]) successfully deleted
I20260927 00:44:55.050289 8490 raft_consensus.cc:482] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:55.050426 8490 raft_consensus.cc:504] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.050846 8490 leader_election.cc:288] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:55.051295 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e3a4f4b32240dea9f7b3081a92ada3" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564" is_pre_election: true
I20260927 00:44:55.051445 8078 raft_consensus.cc:2454] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 0.
I20260927 00:44:55.051728 8290 leader_election.cc:302] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:55.051977 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e3a4f4b32240dea9f7b3081a92ada3" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5" is_pre_election: true
I20260927 00:44:55.052069 8490 raft_consensus.cc:2781] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:55.052107 8219 raft_consensus.cc:2454] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 0.
I20260927 00:44:55.052129 8490 raft_consensus.cc:482] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:55.052166 8490 raft_consensus.cc:3037] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:55.053397 8490 raft_consensus.cc:504] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.053728 8490 leader_election.cc:288] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 election: Requested vote from peers 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447), 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887)
I20260927 00:44:55.054283 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e3a4f4b32240dea9f7b3081a92ada3" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29716c38851c4adaa4842959042bd564"
I20260927 00:44:55.054420 8078 raft_consensus.cc:3037] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:55.055673 8078 raft_consensus.cc:2454] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 1.
I20260927 00:44:55.056022 8290 leader_election.cc:302] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29716c38851c4adaa4842959042bd564, ea9ad3d6b21841fcb77153d1df462b0e; no voters:
I20260927 00:44:55.056092 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e3a4f4b32240dea9f7b3081a92ada3" candidate_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b225aa8208d41c494c110db83d270d5"
I20260927 00:44:55.056211 8490 raft_consensus.cc:2781] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:55.056226 8219 raft_consensus.cc:3037] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:55.056279 8490 raft_consensus.cc:686] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 LEADER]: Becoming Leader. State: Replica: ea9ad3d6b21841fcb77153d1df462b0e, State: Running, Role: LEADER
I20260927 00:44:55.056399 8490 consensus_queue.cc:237] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } }
I20260927 00:44:55.057292 8219 raft_consensus.cc:2454] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9ad3d6b21841fcb77153d1df462b0e in term 1.
I20260927 00:44:55.058781 7920 catalog_manager.cc:5495] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea9ad3d6b21841fcb77153d1df462b0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37753 } health_report { overall_health: HEALTHY } } }
I20260927 00:44:55.077342 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:55.074204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.98.3:37753)
0927 00:44:55.074309 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:55.074315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:55.074347 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:44:55.074509 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:55.074512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:55.074561 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:55.074646 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:55.074884 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:55.075426 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:55.076268 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:55.076503 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:55.076616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:55.076865 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:55.076946 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:55.077025 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:55.077132 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:44:55.077187 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:44:55.084168 8219 raft_consensus.cc:1261] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Refusing update from remote peer ea9ad3d6b21841fcb77153d1df462b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:55.084168 8078 raft_consensus.cc:1261] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Refusing update from remote peer ea9ad3d6b21841fcb77153d1df462b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:55.084606 8463 consensus_queue.cc:1035] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b225aa8208d41c494c110db83d270d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:55.084916 8463 consensus_queue.cc:1035] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29716c38851c4adaa4842959042bd564" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:55.116663 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:55.116847 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:55.119014 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d214220287a34666a659e326ab1491a0: Sending DeleteTablet for 3 replicas of tablet 52e3a4f4b32240dea9f7b3081a92ada3
I20260927 00:44:55.119645 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 00:44:55.119869 8458 tablet_replica.cc:330] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: stopping tablet replica
I20260927 00:44:55.119860 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260927 00:44:55.120000 8458 raft_consensus.cc:2229] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:55.120114 8458 raft_consensus.cc:2258] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:55.120124 8457 tablet_replica.cc:330] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: stopping tablet replica
I20260927 00:44:55.120249 8457 raft_consensus.cc:2229] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:55.120357 8457 raft_consensus.cc:2258] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:55.120806 8458 ts_tablet_manager.cc:1905] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:55.120986 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e3a4f4b32240dea9f7b3081a92ada3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33996
I20260927 00:44:55.121250 8456 tablet_replica.cc:330] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: stopping tablet replica
I20260927 00:44:55.121366 8456 raft_consensus.cc:2229] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:55.121533 8456 raft_consensus.cc:2258] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:55.122182 8457 ts_tablet_manager.cc:1905] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:55.123274 8458 ts_tablet_manager.cc:1918] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:55.123332 8458 log.cc:1196] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/52e3a4f4b32240dea9f7b3081a92ada3
I20260927 00:44:55.123564 8458 ts_tablet_manager.cc:1939] T 52e3a4f4b32240dea9f7b3081a92ada3 P 29716c38851c4adaa4842959042bd564: Deleting consensus metadata
I20260927 00:44:55.124295 7906 catalog_manager.cc:4841] TS 29716c38851c4adaa4842959042bd564 (127.0.98.1:44447): tablet 52e3a4f4b32240dea9f7b3081a92ada3 (table test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]) successfully deleted
I20260927 00:44:55.124650 8457 ts_tablet_manager.cc:1918] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:55.124730 8457 log.cc:1196] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/52e3a4f4b32240dea9f7b3081a92ada3
I20260927 00:44:55.124980 8457 ts_tablet_manager.cc:1939] T 52e3a4f4b32240dea9f7b3081a92ada3 P 8b225aa8208d41c494c110db83d270d5: Deleting consensus metadata
I20260927 00:44:55.125003 8456 ts_tablet_manager.cc:1905] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:55.125422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7994
I20260927 00:44:55.127637 7909 catalog_manager.cc:4841] TS 8b225aa8208d41c494c110db83d270d5 (127.0.98.2:44887): tablet 52e3a4f4b32240dea9f7b3081a92ada3 (table test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]) successfully deleted
I20260927 00:44:55.128003 8456 ts_tablet_manager.cc:1918] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:44:55.128093 8456 log.cc:1196] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/52e3a4f4b32240dea9f7b3081a92ada3
I20260927 00:44:55.128338 8456 ts_tablet_manager.cc:1939] T 52e3a4f4b32240dea9f7b3081a92ada3 P ea9ad3d6b21841fcb77153d1df462b0e: Deleting consensus metadata
I20260927 00:44:55.129012 7907 catalog_manager.cc:4841] TS ea9ad3d6b21841fcb77153d1df462b0e (127.0.98.3:37753): tablet 52e3a4f4b32240dea9f7b3081a92ada3 (table test-table [id=ee8d94db8eeb473fb41b0933ed2a6261]) successfully deleted
I20260927 00:44:55.136914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8134
I20260927 00:44:55.144599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8274
I20260927 00:44:55.151137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 7890
2026-09-27T00:44:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2855 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:55 dist-test-slave-l5sb krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:55 dist-test-slave-l5sb krb5kdc[8520](info): set up 2 sockets
Sep 27 00:44:55 dist-test-slave-l5sb krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260927 00:44:56.190536 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:56 dist-test-slave-l5sb krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:56.309620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:56.420311 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:56.420593 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:56.420637 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:56.425287 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:56.425348 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:56.425370 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:56.425388 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:56.425450 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:56.431309 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:56.432412 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:56.433511 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:56.442433 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:56.442440 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:56.442473 8533 server_base.cc:1034] running on GCE node
W20260927 00:44:56.442440 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:56.443212 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:56.444268 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:56.445433 8533 hybrid_clock.cc:648] HybridClock initialized: now 1790469896445413 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:44:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1305 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:56 dist-test-slave-l5sb krb5kdc[8546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:56 dist-test-slave-l5sb krb5kdc[8546](info): set up 2 sockets
Sep 27 00:44:56 dist-test-slave-l5sb krb5kdc[8546](info): commencing operation
krb5kdc: starting...
W20260927 00:44:57.507205 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:57 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:57.629905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:57.737754 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:57.738034 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:57.738073 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:57.742172 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:57.742239 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:57.742259 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:57.742276 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:57.742292 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:57.747674 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:57.748756 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:57.749840 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:57.758147 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:57.758193 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:57.758249 8559 server_base.cc:1034] running on GCE node
W20260927 00:44:57.758181 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:57.758868 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:57.759961 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:57.761152 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790469897761131 us; error 59 us; skew 500 ppm
Sep 27 00:44:57 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:57.767925 8559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:57.768855 8559 webserver.cc:460] Webserver started at http://127.0.98.62:43185/ using document root <none> and password file <none>
I20260927 00:44:57.769493 8559 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:57.769593 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:57.769910 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:57.772118 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "3914e95f48224458914cda8926af68f8"
format_stamp: "Formatted at 2026-09-27 00:44:57 on dist-test-slave-l5sb"
server_key: "c3b0bb7819d4bb472da15969de256dbe"
server_key_iv: "fe81372a52f0cc35004b3add3494f694"
server_key_version: "encryptionkey@0"
I20260927 00:44:57.772704 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "3914e95f48224458914cda8926af68f8"
format_stamp: "Formatted at 2026-09-27 00:44:57 on dist-test-slave-l5sb"
server_key: "c3b0bb7819d4bb472da15969de256dbe"
server_key_iv: "fe81372a52f0cc35004b3add3494f694"
server_key_version: "encryptionkey@0"
I20260927 00:44:57.776538 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:44:57.779232 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:57.780594 8559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:44:57.780735 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "3914e95f48224458914cda8926af68f8"
format_stamp: "Formatted at 2026-09-27 00:44:57 on dist-test-slave-l5sb"
server_key: "c3b0bb7819d4bb472da15969de256dbe"
server_key_iv: "fe81372a52f0cc35004b3add3494f694"
server_key_version: "encryptionkey@0"
I20260927 00:44:57.780844 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:57.801151 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:57.804131 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:57.804344 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:57.811733 8559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33785
I20260927 00:44:57.811745 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33785 every 8 connection(s)
I20260927 00:44:57.812706 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:57.815517 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:57.815919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8559
I20260927 00:44:57.816112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:57.816437 392 external_mini_cluster.cc:1428] Setting key e99a915233fe916d078b7343f40f4794
I20260927 00:44:57.822037 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Bootstrap starting.
I20260927 00:44:57.824541 8627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:57.825245 8627 log.cc:824] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:44:57 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:57.827368 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: No bootstrap required, opened a new log
I20260927 00:44:57.831358 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:57.817938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.0.98.62:33785)
0927 00:44:57.818368 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:57.818386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:57.818431 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:44:57.819259 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:57.819281 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:57.819689 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:57.820039 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:57.820730 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:57.822193 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:57.823051 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:57.823727 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:57.823886 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:57.826567 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:57.826571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:57.826584 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:57.826608 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:57.828452 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:57.828935 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:57.828939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:57.828943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:57.829087 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:57.829403 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:57.829405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:57.829406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:57.829754 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:57.829951 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:57.830290 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 00:44:57.830612 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260927 00:44:57.831275 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:57.831528 8627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:57.831611 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3914e95f48224458914cda8926af68f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:57.832072 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:57.832229 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:57.832315 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:57.832413 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:57.833259 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:57.833630 8627 leader_election.cc:302] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3914e95f48224458914cda8926af68f8; no voters:
I20260927 00:44:57.833972 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:57.834066 8632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:57.834331 8632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3914e95f48224458914cda8926af68f8, State: Running, Role: LEADER
I20260927 00:44:57.834687 8632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:57.835187 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:57.836349 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3914e95f48224458914cda8926af68f8. Latest consensus state: current_term: 1 leader_uuid: "3914e95f48224458914cda8926af68f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } } }
I20260927 00:44:57.836476 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:57.836433 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3914e95f48224458914cda8926af68f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } } }
I20260927 00:44:57.836524 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:57.836858 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:57.839900 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:57.845621 8641 catalog_manager.cc:1347] Generated new cluster ID: f147b5f8049042c89369ccd4f19882df
I20260927 00:44:57.845688 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:57.852429 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:57.853307 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:57.858889 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Generated new TSK 0
I20260927 00:44:57.859578 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:57.894553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:58.002619 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:58.002857 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:58.002897 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:58.007483 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:58.007546 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:58.007602 8653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:58.012591 8653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:58.013516 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:58.014458 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:58.023805 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.023837 8653 server_base.cc:1034] running on GCE node
W20260927 00:44:58.023800 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.023797 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.024372 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:58.025018 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:58.026196 8653 hybrid_clock.cc:648] HybridClock initialized: now 1790469898026177 us; error 27 us; skew 500 ppm
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:58.029002 8653 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:58.029918 8653 webserver.cc:460] Webserver started at http://127.0.98.1:33187/ using document root <none> and password file <none>
I20260927 00:44:58.030459 8653 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:58.030575 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:58.030800 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:58.032748 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ded6d0a44a04314895674cdb28dfd29"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "6b657bbed2ccd73300290374dd027fa3"
server_key_iv: "4b36c8bf5a6c0c68ee84bc13fbc27f81"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.033299 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ded6d0a44a04314895674cdb28dfd29"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "6b657bbed2ccd73300290374dd027fa3"
server_key_iv: "4b36c8bf5a6c0c68ee84bc13fbc27f81"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.037074 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:44:58.039623 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.040844 8653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:58.040984 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "4ded6d0a44a04314895674cdb28dfd29"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "6b657bbed2ccd73300290374dd027fa3"
server_key_iv: "4b36c8bf5a6c0c68ee84bc13fbc27f81"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.041095 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:58.060083 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:58.063076 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:58.063290 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:58.063902 8653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:58.064730 8653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:58.064802 8653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.064864 8653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:58.064913 8653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.073980 8653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45035
I20260927 00:44:58.074009 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45035 every 8 connection(s)
I20260927 00:44:58.074934 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:58.080473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8653
I20260927 00:44:58.080615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:58.080956 392 external_mini_cluster.cc:1428] Setting key 414f5194f8e6fd192a03295ef7285589
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:58.089865 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.076753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37101 (local address 127.0.98.62:33785)
0927 00:44:58.076902 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:58.076907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:58.077820 (+ 913us) server_negotiation.cc:407] Connection header received
0927 00:44:58.079400 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:58.079403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:58.079461 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:58.079538 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:58.081004 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.081713 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.083072 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.083338 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.083463 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.086267 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:58.086270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:58.086273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:58.086304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:58.087768 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.088296 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.088300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.088302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.088383 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.088727 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.088730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.088731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.088913 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:58.089018 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:58.089515 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 00:44:58.089662 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:58.090715 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.077209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:37101)
0927 00:44:58.077655 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:58.077698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:58.079147 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:58.079756 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:58.079763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:58.080130 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:58.080834 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.080855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.081885 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.081888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.082941 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.082951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.083503 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.083507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.083690 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.084297 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:58.084313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:58.086087 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:58.087885 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.087891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.087907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.088188 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.088473 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.088478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.088480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.088623 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.089017 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:58.089024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:58.089278 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 00:44:58.089464 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:44:58.089822 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260927 00:44:58.091961 8782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33785
I20260927 00:44:58.092257 8782 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:58.092844 8782 heartbeater.cc:511] Master 127.0.98.62:33785 requested a full tablet report, sending...
I20260927 00:44:58.094501 8591 ts_manager.cc:194] Registered new tserver with Master: 4ded6d0a44a04314895674cdb28dfd29 (127.0.98.1:45035)
I20260927 00:44:58.095634 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:58.108011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:58.216514 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:58.216777 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:58.216820 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:58.220815 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:58.220870 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:58.220927 8787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:58.226043 8787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:58.226977 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:58.227946 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:44:58.237706 8787 server_base.cc:1034] running on GCE node
W20260927 00:44:58.237659 8792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.237687 8793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.237946 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.238351 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:58.238952 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:58.240134 8787 hybrid_clock.cc:648] HybridClock initialized: now 1790469898240091 us; error 55 us; skew 500 ppm
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:58.243162 8787 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:58.244091 8787 webserver.cc:460] Webserver started at http://127.0.98.2:41293/ using document root <none> and password file <none>
I20260927 00:44:58.244663 8787 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:58.244745 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:58.245007 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:58.247063 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "1fe90ad3554b41b3b2181a9598ddec9b"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "94317759dc1fdc99cc6815c82ae5c87c"
server_key_iv: "2feb1e255620ef880330e396e6a433f8"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.247581 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1fe90ad3554b41b3b2181a9598ddec9b"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "94317759dc1fdc99cc6815c82ae5c87c"
server_key_iv: "2feb1e255620ef880330e396e6a433f8"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.251348 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:44:58.253880 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.255247 8787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:58.255354 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "1fe90ad3554b41b3b2181a9598ddec9b"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "94317759dc1fdc99cc6815c82ae5c87c"
server_key_iv: "2feb1e255620ef880330e396e6a433f8"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.255448 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:58.273703 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:58.276898 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:58.277060 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:58.277585 8787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:58.278445 8787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:58.278491 8787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.278532 8787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:58.278548 8787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.287853 8787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40375
I20260927 00:44:58.287920 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40375 every 8 connection(s)
I20260927 00:44:58.288784 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:58.294094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8787
I20260927 00:44:58.294252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:58.294600 392 external_mini_cluster.cc:1428] Setting key be1b5d73f635f6b3e6423fe200cfe256
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:58.304427 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.290613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46531 (local address 127.0.98.62:33785)
0927 00:44:58.290768 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:58.290773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:58.291684 (+ 911us) server_negotiation.cc:407] Connection header received
0927 00:44:58.292976 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:58.292979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:58.293025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:58.293101 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:58.294335 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.295082 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.296641 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.296862 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.296944 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.299952 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:58.299958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:58.299965 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:58.299998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:58.302007 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.302658 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.302664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.302668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.302772 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.303149 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.303152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.303154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.303415 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:58.303525 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:58.304054 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 00:44:58.304189 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:44:58.305195 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.291078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:46531)
0927 00:44:58.291516 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:58.291557 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:58.292786 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:58.293266 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:58.293272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:58.293631 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:58.294164 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.294178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.295217 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.295220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.296491 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.296507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.297012 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.297018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.297255 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.298015 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:58.298034 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:58.299775 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:58.302198 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.302204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.302219 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.302545 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.302893 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.302896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.302898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.303044 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.303525 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:58.303530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:58.303792 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 00:44:58.303956 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 00:44:58.304218 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260927 00:44:58.306519 8916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33785
I20260927 00:44:58.306766 8916 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:58.307261 8916 heartbeater.cc:511] Master 127.0.98.62:33785 requested a full tablet report, sending...
I20260927 00:44:58.308382 8591 ts_manager.cc:194] Registered new tserver with Master: 1fe90ad3554b41b3b2181a9598ddec9b (127.0.98.2:40375)
I20260927 00:44:58.308969 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:58.324920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:58.434813 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:58.435067 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:58.435112 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:58.439430 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:58.439495 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:58.439556 8921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:58.444907 8921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:58.446087 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:58.447045 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:58.456180 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.456185 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.456202 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.456461 8921 server_base.cc:1034] running on GCE node
I20260927 00:44:58.456972 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:58.457580 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:58.458763 8921 hybrid_clock.cc:648] HybridClock initialized: now 1790469898458759 us; error 60 us; skew 500 ppm
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:58.461627 8921 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:58.462502 8921 webserver.cc:460] Webserver started at http://127.0.98.3:32773/ using document root <none> and password file <none>
I20260927 00:44:58.463006 8921 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:58.463063 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:58.463223 8921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:58.465121 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b22e2c5dafb401294f2b2f817fe2931"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "e1106262e8432632dfc4d6614bdd0bb3"
server_key_iv: "f1d1f6c983bf8dad2f2cb1adba45fcad"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.465611 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b22e2c5dafb401294f2b2f817fe2931"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "e1106262e8432632dfc4d6614bdd0bb3"
server_key_iv: "f1d1f6c983bf8dad2f2cb1adba45fcad"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.469123 8921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:58.471459 8936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.472642 8921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:58.472730 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "3b22e2c5dafb401294f2b2f817fe2931"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "e1106262e8432632dfc4d6614bdd0bb3"
server_key_iv: "f1d1f6c983bf8dad2f2cb1adba45fcad"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.472808 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:58.482676 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:58.485411 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:58.485574 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:58.486131 8921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:58.486838 8921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:58.486881 8921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.486919 8921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:58.486940 8921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.495252 8921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42461
I20260927 00:44:58.495275 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42461 every 8 connection(s)
I20260927 00:44:58.496115 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:58.500880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 8921
I20260927 00:44:58.500972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:58.501220 392 external_mini_cluster.cc:1428] Setting key cb3a4848c2690c18f5eefc4b61f72199
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:58.509420 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.497880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56565 (local address 127.0.98.62:33785)
0927 00:44:58.498006 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:58.498010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:58.498826 (+ 816us) server_negotiation.cc:407] Connection header received
0927 00:44:58.500180 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:58.500183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:58.500229 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:58.500302 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:58.501544 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.502236 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.503689 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.503896 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.503975 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.506412 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:58.506415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:58.506417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:58.506441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:58.507682 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.508118 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.508120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.508122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.508193 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:58.508497 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:58.508499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:58.508500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:58.508662 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:58.508735 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:58.509200 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:44:58.509304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:44:58.510190 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:58.498247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:56565)
0927 00:44:58.498668 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:58.498709 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:58.499964 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:58.500457 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:58.500463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:58.500816 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:58.501395 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.501411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.502351 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.502371 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.503581 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:58.503590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:58.504016 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:58.504018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:58.504180 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:58.504787 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:58.504803 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:58.506275 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:58.507784 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.507788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.507799 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.508029 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.508269 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:58.508271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:58.508273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:58.508394 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:58.508746 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:58.508751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:58.508959 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 00:44:58.509135 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:44:58.509362 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260927 00:44:58.511305 9050 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33785
I20260927 00:44:58.511533 9050 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:58.511976 9050 heartbeater.cc:511] Master 127.0.98.62:33785 requested a full tablet report, sending...
I20260927 00:44:58.512933 8591 ts_manager.cc:194] Registered new tserver with Master: 3b22e2c5dafb401294f2b2f817fe2931 (127.0.98.3:42461)
I20260927 00:44:58.513449 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56565
I20260927 00:44:58.514956 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:44:58.587728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8653
I20260927 00:44:58.593111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8787
I20260927 00:44:58.598503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8921
I20260927 00:44:58.603722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 8559
I20260927 00:44:58.610146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33785
--webserver_interface=127.0.98.62
--webserver_port=43185
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:58.717607 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:58.717885 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:58.717923 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:58.721906 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:58.721957 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:58.721977 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:58.721993 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:58.722056 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:58.727137 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:58.728132 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:58.729120 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:58.737687 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.737705 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.737855 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.738003 9059 server_base.cc:1034] running on GCE node
I20260927 00:44:58.738571 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:58.739621 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:58.740839 9059 hybrid_clock.cc:648] HybridClock initialized: now 1790469898740801 us; error 45 us; skew 500 ppm
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:58.743788 9059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:58.744715 9059 webserver.cc:460] Webserver started at http://127.0.98.62:43185/ using document root <none> and password file <none>
I20260927 00:44:58.745309 9059 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:58.745427 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:58.749607 9059 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:44:58.751942 9074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.753348 9059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:58.753502 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "3914e95f48224458914cda8926af68f8"
format_stamp: "Formatted at 2026-09-27 00:44:57 on dist-test-slave-l5sb"
server_key: "c3b0bb7819d4bb472da15969de256dbe"
server_key_iv: "fe81372a52f0cc35004b3add3494f694"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.754158 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:58.767707 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:58.770447 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:58.770653 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:58.779449 9059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33785
I20260927 00:44:58.779461 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33785 every 8 connection(s)
I20260927 00:44:58.780458 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:58.784612 9127 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:44:58.785636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9059
I20260927 00:44:58.786085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45035
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33785
--webserver_port=33187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:58.786329 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Bootstrap starting.
I20260927 00:44:58.802488 9127 log.cc:824] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:58.809080 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:58.809463 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Bootstrap complete.
I20260927 00:44:58.813619 9127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:58.814116 9127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3914e95f48224458914cda8926af68f8, State: Initialized, Role: FOLLOWER
I20260927 00:44:58.814646 9127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:58.814790 9127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:58.814878 9127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:58.814997 9127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:44:58.816875 9127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:58.817209 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3914e95f48224458914cda8926af68f8; no voters:
I20260927 00:44:58.817515 9127 leader_election.cc:288] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:44:58.817648 9131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:44:58.817919 9131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [term 2 LEADER]: Becoming Leader. State: Replica: 3914e95f48224458914cda8926af68f8, State: Running, Role: LEADER
I20260927 00:44:58.818292 9131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } }
I20260927 00:44:58.818727 9127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:58.819620 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3914e95f48224458914cda8926af68f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } } }
I20260927 00:44:58.819732 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:58.820399 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3914e95f48224458914cda8926af68f8. Latest consensus state: current_term: 2 leader_uuid: "3914e95f48224458914cda8926af68f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3914e95f48224458914cda8926af68f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33785 } } }
I20260927 00:44:58.820479 9140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:58.820489 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:58.823032 9140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:58.823575 9140 catalog_manager.cc:1259] Loaded cluster ID: f147b5f8049042c89369ccd4f19882df
I20260927 00:44:58.823658 9140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:58.825290 9140 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:58.825903 9140 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3914e95f48224458914cda8926af68f8: Loaded TSK: 0
I20260927 00:44:58.826798 9140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:44:58.917052 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:58.917321 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:58.917366 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:58.921990 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:58.922052 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:58.922113 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:58.927577 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33187
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:58.928666 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:58.929677 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:58.940533 9155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.940500 9154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:58.940537 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:58.941053 9129 server_base.cc:1034] running on GCE node
I20260927 00:44:58.941504 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:58.942283 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:58.943481 9129 hybrid_clock.cc:648] HybridClock initialized: now 1790469898943458 us; error 31 us; skew 500 ppm
Sep 27 00:44:58 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469898, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:58.946898 9129 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:44:58.947999 9129 webserver.cc:460] Webserver started at http://127.0.98.1:33187/ using document root <none> and password file <none>
I20260927 00:44:58.948696 9129 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:58.948802 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:58.953720 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:58.956141 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:58.957628 9129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:58.957762 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "4ded6d0a44a04314895674cdb28dfd29"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "6b657bbed2ccd73300290374dd027fa3"
server_key_iv: "4b36c8bf5a6c0c68ee84bc13fbc27f81"
server_key_version: "encryptionkey@0"
I20260927 00:44:58.958506 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:58.996857 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:59.000334 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:59.000547 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:59.001178 9129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:59.002071 9129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:59.002128 9129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.002169 9129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:59.002185 9129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.011194 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45035
I20260927 00:44:59.011216 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45035 every 8 connection(s)
I20260927 00:44:59.012204 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:59.012832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9129
I20260927 00:44:59.013316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40375
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33785
--webserver_port=41293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.027262 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.014696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:53835)
0927 00:44:59.015245 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.015298 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.016544 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.017985 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.017992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.018355 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.018903 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.018918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.021171 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.021174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.022133 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.022142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.023252 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.023255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.023385 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.024038 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.024055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.026452 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":109,"threads_started":1}
W20260927 00:44:59.027536 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.014603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53835 (local address 127.0.98.62:33785)
0927 00:44:59.015152 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.015174 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.015420 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:44:59.016995 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.017015 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.017467 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.017894 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.019130 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.021021 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.022253 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.023084 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.023293 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.026700 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53835: BlockingRecv error: recv got EOF from 127.0.98.1:53835 (error 108)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":135,"threads_started":1}
W20260927 00:44:59.027810 9278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33785 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33785: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.033344 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.028281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:56189)
0927 00:44:59.028389 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.028404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.028480 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.028719 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.028721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.028843 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.028926 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.028930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.029928 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.029931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.030468 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.030475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.031106 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.031109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.031176 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.031640 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.031651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.033256 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:44:59.033510 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.028316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56189 (local address 127.0.98.62:33785)
0927 00:44:59.028430 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.028435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.028449 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:59.028566 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.028569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.028615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.028698 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.029021 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.029836 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.030572 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.030976 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.031088 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.033402 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56189: BlockingRecv error: recv got EOF from 127.0.98.1:56189 (error 108)
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":57}
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.038544 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.034024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:45413)
0927 00:44:59.034131 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.034148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.034221 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.034456 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.034458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.034601 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.034662 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.034666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.035347 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.035353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.035852 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.035858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.036297 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.036300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.036375 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.036793 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.036803 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.038414 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 00:44:59.038630 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.034097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45413 (local address 127.0.98.62:33785)
0927 00:44:59.034202 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.034206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.034218 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:44:59.034313 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.034315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.034353 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.034443 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.034769 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.035245 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.035965 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.036193 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.036299 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.038518 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45413: BlockingRecv error: recv got EOF from 127.0.98.1:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 00:44:59.038931 9278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:44:59.126423 9280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:59.126698 9280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:59.126773 9280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:59.130836 9280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:59.130909 9280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:59.131006 9280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:59.136406 9280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41293
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:59.137353 9280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:59.138365 9280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:59.147370 9287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:59.147346 9288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:59.147346 9290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:59.147401 9280 server_base.cc:1034] running on GCE node
I20260927 00:44:59.147956 9280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:59.148561 9280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:59.149748 9280 hybrid_clock.cc:648] HybridClock initialized: now 1790469899149724 us; error 47 us; skew 500 ppm
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:59.152407 9280 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:44:59.153277 9280 webserver.cc:460] Webserver started at http://127.0.98.2:41293/ using document root <none> and password file <none>
I20260927 00:44:59.153878 9280 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:59.153981 9280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:59.157665 9280 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:59.159785 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.160966 9280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:59.161099 9280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "1fe90ad3554b41b3b2181a9598ddec9b"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "94317759dc1fdc99cc6815c82ae5c87c"
server_key_iv: "2feb1e255620ef880330e396e6a433f8"
server_key_version: "encryptionkey@0"
I20260927 00:44:59.161621 9280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:59.188930 9280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:59.192255 9280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:59.192459 9280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:59.193053 9280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:59.193887 9280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:59.193970 9280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.194042 9280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:59.194085 9280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.203176 9280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40375
I20260927 00:44:59.203202 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40375 every 8 connection(s)
I20260927 00:44:59.204167 9280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:59.209690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9280
I20260927 00:44:59.210163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42461
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33785
--webserver_port=32773
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:44:59.215427 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.205886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47087 (local address 127.0.98.62:33785)
0927 00:44:59.206023 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.206027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.206857 (+ 830us) server_negotiation.cc:407] Connection header received
0927 00:44:59.208218 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.208222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.208285 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.208394 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.209535 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.210154 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.211277 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.211502 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.211622 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.215326 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47087: BlockingRecv error: recv got EOF from 127.0.98.2:47087 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:59.216333 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.206281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:47087)
0927 00:44:59.206707 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.206748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.207996 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.208575 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.208583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.208922 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.209384 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.209397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.210279 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.210284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.211147 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.211156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.211651 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.211655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.211846 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.212458 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.212474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.215212 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
W20260927 00:44:59.216809 9411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33785 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33785: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.222518 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.217275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:53567)
0927 00:44:59.217390 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.217406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.217487 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.217739 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.217742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.217915 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.218002 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.218007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.218839 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.218844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.219510 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.219519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.220009 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.220013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.220118 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.220608 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.220621 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.222394 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:44:59.222734 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.217365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53567 (local address 127.0.98.62:33785)
0927 00:44:59.217476 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.217479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.217493 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:44:59.217591 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.217593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.217643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.217725 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.218139 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.218733 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.219625 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.219854 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.220011 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.222506 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53567: BlockingRecv error: recv got EOF from 127.0.98.2:53567 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.228073 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.222989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:57285)
0927 00:44:59.223129 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.223142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.223205 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.223417 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.223420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.223576 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.223670 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.223676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.224378 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.224382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.225252 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.225262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.225653 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.225657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.225738 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.226195 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.226209 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.227940 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 00:44:59.228142 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.223053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57285 (local address 127.0.98.62:33785)
0927 00:44:59.223169 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.223172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.223183 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:59.223282 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.223284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.223316 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.223378 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.223787 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.224281 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.225364 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.225556 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.225625 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.228041 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57285: BlockingRecv error: recv got EOF from 127.0.98.2:57285 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:44:59.228390 9411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:44:59.322186 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:59.322435 9414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:59.322515 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:59.326496 9414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:59.326547 9414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:59.326648 9414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:59.331715 9414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32773
--tserver_master_addrs=127.0.98.62:33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:44:59.332686 9414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:59.333668 9414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:59.343356 9419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:59.343446 9414 server_base.cc:1034] running on GCE node
W20260927 00:44:59.343386 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:59.343356 9420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:59.344373 9414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:59.344983 9414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:59.346161 9414 hybrid_clock.cc:648] HybridClock initialized: now 1790469899346132 us; error 49 us; skew 500 ppm
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:59.348773 9414 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:44:59.349628 9414 webserver.cc:460] Webserver started at http://127.0.98.3:32773/ using document root <none> and password file <none>
I20260927 00:44:59.350222 9414 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:59.350335 9414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:59.354012 9414 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:44:59.356017 9429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.357203 9414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:59.357348 9414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "3b22e2c5dafb401294f2b2f817fe2931"
format_stamp: "Formatted at 2026-09-27 00:44:58 on dist-test-slave-l5sb"
server_key: "e1106262e8432632dfc4d6614bdd0bb3"
server_key_iv: "f1d1f6c983bf8dad2f2cb1adba45fcad"
server_key_version: "encryptionkey@0"
I20260927 00:44:59.357945 9414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:59.372354 9414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:59.375046 9414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:59.375243 9414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:59.375800 9414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:59.376591 9414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:59.376662 9414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.376727 9414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:59.376773 9414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:59.385614 9414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42461
I20260927 00:44:59.385643 9542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42461 every 8 connection(s)
I20260927 00:44:59.386627 9414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.396507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9414
W20260927 00:44:59.397444 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.388314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38719 (local address 127.0.98.62:33785)
0927 00:44:59.388478 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.388483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.389303 (+ 820us) server_negotiation.cc:407] Connection header received
0927 00:44:59.390671 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.390673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.390719 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.390792 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.391918 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.392430 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.393543 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.393762 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.393935 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.397315 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38719: BlockingRecv error: recv got EOF from 127.0.98.3:38719 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:44:59.398168 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.388721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:38719)
0927 00:44:59.389154 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.389195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.390481 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.390952 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.390958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.391324 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.391784 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.391796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.392533 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.392536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.393404 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.393412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.393986 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.393989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.394146 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.394765 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.394780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.397135 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
W20260927 00:44:59.398655 9543 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33785 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33785: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.404716 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.399069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:60437)
0927 00:44:59.399191 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.399205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.399277 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.399580 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.399583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.399739 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.399864 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.399869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.400893 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.400896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.401626 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.401634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.402183 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.402187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.402284 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.402683 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.402699 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.404632 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 00:44:59.404914 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.399178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60437 (local address 127.0.98.62:33785)
0927 00:44:59.399397 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.399400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.399411 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:59.399437 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.399439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.399469 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.399538 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.399965 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.400757 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.401731 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.402070 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.402148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.404786 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60437: BlockingRecv error: recv got EOF from 127.0.98.3:60437 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260927 00:44:59.407684 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.397739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.0.98.62:33785)
0927 00:44:59.397877 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.397880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.397892 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:44:59.397922 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.397924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.397962 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.398033 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.398919 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.399698 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.400904 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.401272 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.401374 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.403055 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:59.403060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:59.403075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:59.403136 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:59.405632 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:59.406107 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:59.406112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:59.406116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:59.406231 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:59.406512 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:59.406517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:59.406518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:59.406857 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:59.407070 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:59.407320 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:44:59.407547 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:44:59 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:44:59.410207 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:59.405230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:43323)
0927 00:44:59.405315 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:59.405347 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:59.405420 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:59.405629 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:59.405631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:59.405809 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:59.405884 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.405889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.406585 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.406588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.407138 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:59.407150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.407711 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.407717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:59.407807 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.408245 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:59.408258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:59.410062 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 00:44:59.410444 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:59.405275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43323 (local address 127.0.98.62:33785)
0927 00:44:59.405432 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:59.405436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:59.405451 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:44:59.405488 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:59.405492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:59.405525 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:59.405616 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:59.405989 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.406469 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.407272 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:59.407600 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:59.407672 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:59.410203 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43323: BlockingRecv error: recv got EOF from 127.0.98.3:43323 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 00:44:59.410517 9543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:00.045403 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.039605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:44321)
0927 00:45:00.039895 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.039908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:00.039996 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:00.040272 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:00.040274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:00.040451 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:00.040568 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.040574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.041359 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.041362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.041970 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.041978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.042467 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.042469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.042541 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.043037 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:00.043048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:00.045244 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
W20260927 00:45:00.045524 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:00.039656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44321 (local address 127.0.98.62:33785)
0927 00:45:00.039937 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:00.039940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:00.039954 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:00.040113 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:00.040116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:00.040158 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:00.040237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:00.040699 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.041252 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.042128 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.042359 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.042450 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.045374 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44321: BlockingRecv error: recv got EOF from 127.0.98.1:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:00.234738 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.229119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:33951)
0927 00:45:00.229403 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.229416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:00.229544 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:00.229855 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:00.229858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:00.230076 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:00.230198 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.230204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.230981 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.230988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.231611 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.231620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.232089 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.232092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.232198 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.232698 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:00.232710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:00.234611 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
W20260927 00:45:00.234848 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:00.229221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33951 (local address 127.0.98.62:33785)
0927 00:45:00.229370 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:00.229374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:00.229500 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:45:00.229642 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:00.229644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:00.229690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:00.229818 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:00.230331 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.230864 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.231739 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.231965 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.232079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.234732 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33951: BlockingRecv error: recv got EOF from 127.0.98.2:33951 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:00 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:00.416703 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.411185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:53233)
0927 00:45:00.411459 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.411473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:00.411586 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:00.411825 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:00.411827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:00.411996 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:00.412090 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.412094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.412804 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.412807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.413492 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:00.413502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.414050 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.414060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:00.414131 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.414717 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:00.414729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:00.416559 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
W20260927 00:45:00.416829 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:00.411316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53233 (local address 127.0.98.62:33785)
0927 00:45:00.411484 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:00.411488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:00.411544 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:45:00.411673 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:00.411676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:00.411720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:00.411788 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:00.412188 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.412710 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.413579 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:00.413931 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:00.414050 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:00.416706 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53233: BlockingRecv error: recv got EOF from 127.0.98.3:53233 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:01.052119 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.046263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:42477)
0927 00:45:01.046530 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.046550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:01.046638 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:01.046872 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:01.046874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:01.047054 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:01.047179 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.047185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.047934 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.047937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.048495 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.048504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.048939 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.048942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.049007 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.049523 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:01.049534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:01.051974 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
W20260927 00:45:01.052206 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:01.046380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42477 (local address 127.0.98.62:33785)
0927 00:45:01.046621 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:01.046624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:01.046637 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:01.046723 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:01.046725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:01.046766 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:01.046836 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:01.047285 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.047809 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.048617 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.048837 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.048917 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.052104 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42477: BlockingRecv error: recv got EOF from 127.0.98.1:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:01.241504 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.235690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:36555)
0927 00:45:01.235961 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.236012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:01.236116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:01.236405 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:01.236408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:01.236665 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:01.236793 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.236808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.237597 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.237602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.238259 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.238268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.238734 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.238737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.238824 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.239369 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:01.239380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:01.241311 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
W20260927 00:45:01.241568 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:01.235764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36555 (local address 127.0.98.62:33785)
0927 00:45:01.235927 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:01.235930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:01.236098 (+ 168us) server_negotiation.cc:407] Connection header received
0927 00:45:01.236206 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:01.236209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:01.236264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:01.236366 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:01.236923 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.237482 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.238383 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.238623 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.238730 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.241455 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36555: BlockingRecv error: recv got EOF from 127.0.98.2:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:01 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:01.423064 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.417604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:33603)
0927 00:45:01.417908 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.417920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:01.418014 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:01.418231 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:01.418233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:01.418403 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:01.418492 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.418496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.419334 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.419337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.419889 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:01.419896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.420362 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.420365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:01.420435 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.420989 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:01.421001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:01.422892 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":116,"threads_started":1}
W20260927 00:45:01.423307 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:01.417722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33603 (local address 127.0.98.62:33785)
0927 00:45:01.417890 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:01.417893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:01.418026 (+ 133us) server_negotiation.cc:407] Connection header received
0927 00:45:01.418097 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:01.418100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:01.418145 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:01.418213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:01.418618 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.419233 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.419983 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:01.420260 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:01.420342 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:01.423042 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33603: BlockingRecv error: recv got EOF from 127.0.98.3:33603 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:02.059432 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.052974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:58861)
0927 00:45:02.053263 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.053276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:02.053364 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:02.053712 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:02.053715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:02.053930 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:02.054089 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.054095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.054917 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.054920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.055571 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.055581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.056144 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.056147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.056223 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.056785 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:02.056797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:02.059252 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
W20260927 00:45:02.059662 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:02.053177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58861 (local address 127.0.98.62:33785)
0927 00:45:02.053451 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:02.053455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:02.053469 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:02.053509 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:02.053511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:02.053554 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:02.053645 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:02.054235 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.054782 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.055698 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.055997 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.056091 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.059442 (+ 3351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58861: BlockingRecv error: recv got EOF from 127.0.98.1:58861 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":115,"threads_started":1}
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:02.248413 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.242340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:54995)
0927 00:45:02.242611 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.242626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:02.242730 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:02.243196 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:02.243199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:02.243380 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:02.243481 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.243487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.244421 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.244424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.245082 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.245094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.245653 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.245657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.245749 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.246394 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:02.246407 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:02.248241 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
W20260927 00:45:02.248468 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:02.242470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54995 (local address 127.0.98.62:33785)
0927 00:45:02.242649 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:02.242653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:02.242691 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:45:02.242866 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:02.242869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:02.242917 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:02.242989 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:02.243658 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.244294 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.245242 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.245526 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.245647 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.248361 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54995: BlockingRecv error: recv got EOF from 127.0.98.2:54995 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:02 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:02.429318 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.423788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:42451)
0927 00:45:02.424066 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.424078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:02.424210 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:02.424478 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:02.424480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:02.424690 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:02.424776 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.424782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.425544 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.425571 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.426291 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.426301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.426739 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.426743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.426822 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.427356 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:02.427368 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:02.429120 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
W20260927 00:45:02.429337 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:02.423859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42451 (local address 127.0.98.62:33785)
0927 00:45:02.423988 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:02.423992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:02.424149 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:45:02.424316 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:02.424326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:02.424371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:02.424453 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:02.424889 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.425436 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.426404 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.426630 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.426737 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.429239 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42451: BlockingRecv error: recv got EOF from 127.0.98.3:42451 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:03.071270 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.060260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:35653)
0927 00:45:03.060614 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.060628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:03.060716 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:03.061002 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:03.061007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:03.061195 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:03.061356 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.061363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.062178 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.062180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.062803 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.062812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.063251 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.063254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.063317 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.069231 (+ 5914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:03.069244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:03.071153 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
W20260927 00:45:03.071429 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:03.060406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35653 (local address 127.0.98.62:33785)
0927 00:45:03.060664 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:03.060667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:03.060680 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:03.060805 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:03.060807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:03.060850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:03.060934 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:03.061459 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.062041 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.062902 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.063154 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.063230 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.071292 (+ 8062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35653: BlockingRecv error: recv got EOF from 127.0.98.1:35653 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":58,"threads_started":1}
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:03.254766 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.249369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:54209)
0927 00:45:03.249651 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.249664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:03.249759 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:03.250017 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:03.250020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:03.250195 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:03.250289 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.250294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.250982 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.250984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.251516 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.251523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.251908 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.251910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.251968 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.252591 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:03.252603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:03.254652 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
W20260927 00:45:03.254935 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:03.249434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54209 (local address 127.0.98.62:33785)
0927 00:45:03.249582 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:03.249586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:03.249736 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:45:03.249860 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:03.249862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:03.249904 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:03.249972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:03.250386 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.250894 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.251613 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.251817 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.251923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.254779 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54209: BlockingRecv error: recv got EOF from 127.0.98.2:54209 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:03 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:03.435992 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.430237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:55401)
0927 00:45:03.430507 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.430522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:03.430621 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:03.430861 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:03.430863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:03.431084 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:03.431181 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.431187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.431923 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.431927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.432516 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:03.432526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.432946 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.432949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:03.433066 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.433680 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:03.433693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:03.435876 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
W20260927 00:45:03.436168 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:03.430357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55401 (local address 127.0.98.62:33785)
0927 00:45:03.430493 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:03.430496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:03.430593 (+ 97us) server_negotiation.cc:407] Connection header received
0927 00:45:03.430706 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:03.430709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:03.430753 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:03.430824 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:03.431283 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.431803 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.432627 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:03.432843 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:03.432921 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:03.436029 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55401: BlockingRecv error: recv got EOF from 127.0.98.3:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:04.077646 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.072178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:54917)
0927 00:45:04.072443 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.072456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:04.072540 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:04.072786 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:04.072788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:04.072964 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:04.073082 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.073088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.073853 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.073856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.074433 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.074441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.075040 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.075044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.075142 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.075730 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:04.075743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:04.077521 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260927 00:45:04.077775 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:04.072297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54917 (local address 127.0.98.62:33785)
0927 00:45:04.072530 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:04.072533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:04.072546 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:04.072630 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:04.072632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:04.072672 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:04.072742 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:04.073185 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.073704 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.074552 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.074871 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.074996 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.077646 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54917: BlockingRecv error: recv got EOF from 127.0.98.1:54917 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:04.262235 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.255538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:41867)
0927 00:45:04.255871 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.255888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:04.256035 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:04.256310 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:04.256313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:04.256533 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:04.256628 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.256634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.257439 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.257444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.258503 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.258523 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.259183 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.259188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.259304 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.259932 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:04.259947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:04.262048 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":116,"threads_started":1}
W20260927 00:45:04.262307 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:04.255744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41867 (local address 127.0.98.62:33785)
0927 00:45:04.255890 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:04.255894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:04.255974 (+ 80us) server_negotiation.cc:407] Connection header received
0927 00:45:04.256143 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:04.256147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:04.256200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:04.256286 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:04.256748 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.257326 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.258624 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.259048 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.259157 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.262169 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41867: BlockingRecv error: recv got EOF from 127.0.98.2:41867 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:04 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:04.442204 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.436779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:55121)
0927 00:45:04.437078 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.437090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:04.437183 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:04.437395 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:04.437397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:04.437568 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:04.437656 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.437661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.438437 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.438439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.439003 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:04.439018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.439414 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.439416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:04.439474 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.439945 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:04.439955 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:04.442020 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":135,"threads_started":1}
W20260927 00:45:04.442245 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:04.436861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55121 (local address 127.0.98.62:33785)
0927 00:45:04.437036 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:04.437043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:04.437192 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:45:04.437244 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:04.437246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:04.437294 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:04.437367 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:04.437759 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.438333 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.439108 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:04.439322 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:04.439416 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:04.442138 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55121: BlockingRecv error: recv got EOF from 127.0.98.3:55121 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:05.084359 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.078558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:57915)
0927 00:45:05.078816 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.078829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:05.078933 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:05.079178 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:05.079180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:05.079358 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:05.079456 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.079461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.080210 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.080212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.080765 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.080773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.081262 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.081265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.081331 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.082005 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:05.082019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:05.084240 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
W20260927 00:45:05.084574 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:05.078643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57915 (local address 127.0.98.62:33785)
0927 00:45:05.078933 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:05.078938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:05.078956 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:45:05.079002 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:05.079004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:05.079056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:05.079140 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:05.079574 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.080110 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.080896 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.081142 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.081233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.084381 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57915: BlockingRecv error: recv got EOF from 127.0.98.1:57915 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:05.268913 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.263211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:41091)
0927 00:45:05.263537 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.263550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:05.263649 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:05.263865 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:05.263867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:05.264045 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:05.264159 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.264164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.265062 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.265065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.265681 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.265690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.266219 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.266223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.266303 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.266892 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:05.266928 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:05.268801 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
W20260927 00:45:05.269104 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:05.263279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41091 (local address 127.0.98.62:33785)
0927 00:45:05.263485 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:05.263490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:05.263621 (+ 131us) server_negotiation.cc:407] Connection header received
0927 00:45:05.263715 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:05.263717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:05.263762 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:05.263830 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:05.264283 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.264956 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.265876 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.266092 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.266171 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.268957 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41091: BlockingRecv error: recv got EOF from 127.0.98.2:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:05 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:05.448302 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.442971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:51971)
0927 00:45:05.443244 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.443260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:05.443387 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:05.443680 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:05.443683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:05.443887 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:05.443976 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.443982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.444765 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.444771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.445347 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:05.445355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.445835 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.445839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:05.445925 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.446399 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:05.446411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:05.448186 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
W20260927 00:45:05.448427 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:05.443049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51971 (local address 127.0.98.62:33785)
0927 00:45:05.443205 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:05.443209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:05.443336 (+ 127us) server_negotiation.cc:407] Connection header received
0927 00:45:05.443515 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:05.443518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:05.443574 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:05.443658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:05.444098 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.444659 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.445469 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:05.445703 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:05.445835 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:05.448311 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51971: BlockingRecv error: recv got EOF from 127.0.98.3:51971 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:06.090647 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.085269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:39405)
0927 00:45:06.085774 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.085815 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:06.085915 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:06.086168 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:06.086170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:06.086349 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:06.086443 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.086448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.087173 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.087175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.087737 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.087745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.088152 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.088154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.088216 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.088705 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:06.088716 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:06.090531 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":90,"threads_started":1}
W20260927 00:45:06.090837 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:06.085381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39405 (local address 127.0.98.62:33785)
0927 00:45:06.085645 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:06.085650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:06.085889 (+ 239us) server_negotiation.cc:407] Connection header received
0927 00:45:06.086000 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:06.086002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:06.086048 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:06.086126 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:06.086554 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.087073 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.087836 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.088056 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.088129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.090665 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39405: BlockingRecv error: recv got EOF from 127.0.98.1:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":59,"threads_started":1}
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:06.274928 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.269676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:47573)
0927 00:45:06.270036 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.270061 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:06.270159 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:06.270389 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:06.270391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:06.270577 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:06.270672 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.270677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.271399 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.271403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.271983 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.271994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.272393 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.272395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.272454 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.272951 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:06.272970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:06.274816 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
W20260927 00:45:06.275084 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:06.269738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47573 (local address 127.0.98.62:33785)
0927 00:45:06.269896 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:06.269899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:06.270131 (+ 232us) server_negotiation.cc:407] Connection header received
0927 00:45:06.270237 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:06.270240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:06.270284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:06.270352 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:06.270773 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.271287 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.272090 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.272298 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.272373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.274947 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47573: BlockingRecv error: recv got EOF from 127.0.98.2:47573 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:06 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:06.455152 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.449206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:47219)
0927 00:45:06.449531 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.449543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:06.449676 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:06.449958 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:06.449960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:06.450173 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:06.450263 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.450269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.451027 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.451033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.451590 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:06.451598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.452056 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.452059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:06.452146 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.452800 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:06.452815 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:06.455037 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
W20260927 00:45:06.455329 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:06.449283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47219 (local address 127.0.98.62:33785)
0927 00:45:06.449451 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:06.449456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:06.449618 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:45:06.449796 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:06.449799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:06.449853 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:06.449938 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:06.450379 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.450908 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.451712 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:06.451938 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:06.452054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:06.455173 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47219: BlockingRecv error: recv got EOF from 127.0.98.3:47219 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:07.096663 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.091401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:53087)
0927 00:45:07.091708 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.091724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:07.091830 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:07.092079 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:07.092081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:07.092256 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:07.092348 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.092354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.093073 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.093075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.093629 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.093636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.094107 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.094109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.094170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.094660 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:07.094671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:07.096539 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":64,"threads_started":1}
W20260927 00:45:07.096798 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:07.091596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53087 (local address 127.0.98.62:33785)
0927 00:45:07.091827 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:07.091831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:07.091843 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:07.091895 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:07.091897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:07.091943 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:07.092037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:07.092449 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.092960 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.093755 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.094012 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.094087 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.096662 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53087: BlockingRecv error: recv got EOF from 127.0.98.1:53087 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:07.281433 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.275806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:36765)
0927 00:45:07.276140 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.276153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:07.276248 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:07.276485 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:07.276486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:07.276666 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:07.276760 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.276765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.277670 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.277673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.278276 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.278286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.278736 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.278740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.278806 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.279383 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:07.279396 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:07.281310 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":151,"threads_started":1}
W20260927 00:45:07.281629 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:07.275861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36765 (local address 127.0.98.62:33785)
0927 00:45:07.276002 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:07.276008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:07.276218 (+ 210us) server_negotiation.cc:407] Connection header received
0927 00:45:07.276314 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:07.276316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:07.276362 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:07.276429 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:07.276857 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.277544 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.278388 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.278626 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.278700 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.281451 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36765: BlockingRecv error: recv got EOF from 127.0.98.2:36765 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:07 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:07.461970 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.455890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:37367)
0927 00:45:07.456175 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.456202 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:07.456306 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:07.456721 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:07.456724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:07.456988 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:07.457090 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.457096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.457923 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.457926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.458632 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:07.458641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.459073 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.459076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:07.459156 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.459650 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:07.459664 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:07.461768 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260927 00:45:07.462010 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:07.456007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37367 (local address 127.0.98.62:33785)
0927 00:45:07.456181 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:07.456186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:07.456274 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:45:07.456388 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:07.456391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:07.456437 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:07.456514 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:07.457234 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.457759 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.458733 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:07.458958 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:07.459037 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:07.461912 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37367: BlockingRecv error: recv got EOF from 127.0.98.3:37367 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:08.108201 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.097601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:53401)
0927 00:45:08.097905 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.097919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:08.098017 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:08.098253 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:08.098255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:08.098431 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:08.098524 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.098529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.099289 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.099292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.099836 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.099842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.100277 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.100279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.100341 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.105923 (+ 5582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:08.105936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:08.108028 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
W20260927 00:45:08.108444 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:08.097691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53401 (local address 127.0.98.62:33785)
0927 00:45:08.097976 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:08.097979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:08.097993 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:08.098100 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:08.098102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:08.098145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:08.098220 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:08.098629 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.099194 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.099933 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.100178 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.100256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.108306 (+ 8050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53401: BlockingRecv error: recv got EOF from 127.0.98.1:53401 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:08.288326 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.282335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:54213)
0927 00:45:08.282696 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.282709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:08.282847 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:08.283143 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:08.283146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:08.283385 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:08.283509 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.283514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.284513 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.284516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.285070 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.285077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.285512 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.285514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.285620 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.286243 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:08.286255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:08.288211 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
W20260927 00:45:08.288522 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:08.282365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54213 (local address 127.0.98.62:33785)
0927 00:45:08.282524 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:08.282529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:08.282810 (+ 281us) server_negotiation.cc:407] Connection header received
0927 00:45:08.282946 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:08.282950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:08.283007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:08.283091 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:08.283620 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.284386 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.285183 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.285410 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.285493 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.288355 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54213: BlockingRecv error: recv got EOF from 127.0.98.2:54213 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:08 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:08.469101 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.462788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:55841)
0927 00:45:08.463188 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.463200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:08.463350 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:08.463636 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:08.463639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:08.463846 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:08.463942 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.463947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.464882 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.464888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.465457 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:08.465465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.466029 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.466033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:08.466119 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.466698 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:08.466710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:08.468985 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":98,"threads_started":1}
W20260927 00:45:08.469236 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:08.462852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55841 (local address 127.0.98.62:33785)
0927 00:45:08.463104 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:08.463109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:08.463282 (+ 173us) server_negotiation.cc:407] Connection header received
0927 00:45:08.463476 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:08.463479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:08.463525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:08.463614 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:08.464061 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.464772 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.465577 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:08.465913 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:08.466026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:08.469120 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55841: BlockingRecv error: recv got EOF from 127.0.98.3:55841 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:09.114365 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.108954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:50803)
0927 00:45:09.109214 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.109227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:09.109309 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:09.109554 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:09.109556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:09.109730 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:09.109852 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.109857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.110572 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.110575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.111124 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.111131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.111568 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.111573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.111652 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.112232 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:09.112249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:09.114253 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
W20260927 00:45:09.114567 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:09.109062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50803 (local address 127.0.98.62:33785)
0927 00:45:09.109301 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:09.109304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:09.109317 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:09.109376 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:09.109378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:09.109418 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:09.109517 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:09.109949 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.110484 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.111220 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.111433 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.111539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.114379 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50803: BlockingRecv error: recv got EOF from 127.0.98.1:50803 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:09.294654 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.289136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:53329)
0927 00:45:09.289395 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.289409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:09.289492 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:09.289786 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:09.289789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:09.289984 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:09.290082 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.290088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.290868 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.290871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.291424 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.291433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.291870 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.291872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.291933 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.292417 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:09.292428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:09.294480 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
W20260927 00:45:09.294723 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:09.289265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53329 (local address 127.0.98.62:33785)
0927 00:45:09.289466 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:09.289469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:09.289483 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:09.289581 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:09.289584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:09.289625 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:09.289698 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:09.290189 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.290725 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.291537 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.291749 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.291826 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.294607 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53329: BlockingRecv error: recv got EOF from 127.0.98.2:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:09 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:09.475520 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.470068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:38731)
0927 00:45:09.470340 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.470353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:09.470449 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:09.470682 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:09.470684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:09.470855 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:09.470988 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.470993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.471744 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.471747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.472316 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:09.472324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.472774 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.472777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:09.472854 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.473336 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:09.473350 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:09.475346 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260927 00:45:09.475592 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:09.470200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38731 (local address 127.0.98.62:33785)
0927 00:45:09.470361 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:09.470364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:09.470423 (+ 59us) server_negotiation.cc:407] Connection header received
0927 00:45:09.470532 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:09.470535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:09.470580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:09.470654 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:09.471091 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.471614 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.472436 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:09.472645 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:09.472725 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:09.475481 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38731: BlockingRecv error: recv got EOF from 127.0.98.3:38731 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:10.120474 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.115106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:45317)
0927 00:45:10.115371 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.115384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:10.115476 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:10.115708 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:10.115710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:10.115885 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:10.115990 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.115995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.116731 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.116734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.117329 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.117338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.117760 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.117762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.117883 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.118456 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:10.118468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:10.120316 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
W20260927 00:45:10.120699 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:10.115175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45317 (local address 127.0.98.62:33785)
0927 00:45:10.115467 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:10.115470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:10.115482 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:10.115559 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:10.115561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:10.115604 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:10.115678 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:10.116085 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.116624 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.117446 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.117665 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.117741 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.120563 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45317: BlockingRecv error: recv got EOF from 127.0.98.1:45317 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:10.301955 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.295735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:43541)
0927 00:45:10.296066 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.296103 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:10.296245 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:10.296583 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:10.296586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:10.296770 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:10.296865 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.296870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.298087 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.298091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.298660 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.298669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.299201 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.299205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.299292 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.299898 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:10.299955 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:10.301761 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
W20260927 00:45:10.302050 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:10.295795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43541 (local address 127.0.98.62:33785)
0927 00:45:10.295991 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:10.295998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:10.296183 (+ 185us) server_negotiation.cc:407] Connection header received
0927 00:45:10.296358 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:10.296361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:10.296405 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:10.296536 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:10.297023 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.297982 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.298773 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.299073 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.299192 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.301900 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43541: BlockingRecv error: recv got EOF from 127.0.98.2:43541 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:10 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:10.481549 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.476274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:60501)
0927 00:45:10.476540 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.476555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:10.476660 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:10.476907 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:10.476909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:10.477087 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:10.477180 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.477185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.477935 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.477938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.478507 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:10.478517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.478976 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.478979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:10.479046 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.479607 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:10.479626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:10.481417 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
W20260927 00:45:10.481702 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:10.476380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60501 (local address 127.0.98.62:33785)
0927 00:45:10.476523 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:10.476526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:10.476629 (+ 103us) server_negotiation.cc:407] Connection header received
0927 00:45:10.476737 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:10.476739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:10.476785 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:10.476853 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:10.477274 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.477830 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.478614 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:10.478849 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:10.478955 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:10.481548 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60501: BlockingRecv error: recv got EOF from 127.0.98.3:60501 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:11.126538 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.121254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:52431)
0927 00:45:11.121503 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.121519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:11.121624 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:11.121978 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:11.121980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:11.122165 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:11.122262 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.122267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.123030 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.123032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.123583 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.123590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.124017 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.124019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.124078 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.124562 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:11.124573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:11.126412 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
W20260927 00:45:11.126752 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:11.121354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52431 (local address 127.0.98.62:33785)
0927 00:45:11.121583 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:11.121586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:11.121617 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:45:11.121748 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:11.121751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:11.121871 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:11.121949 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:11.122370 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.122935 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.123679 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.123925 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.124000 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.126569 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52431: BlockingRecv error: recv got EOF from 127.0.98.1:52431 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:45:11.308339 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:11.302882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40885 (local address 127.0.98.62:33785)
0927 00:45:11.303067 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:11.303071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:11.303120 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:45:11.303232 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:11.303234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:11.303276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:11.303348 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:11.303814 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.304358 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.305134 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.305344 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.305426 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.308201 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40885: BlockingRecv error: recv got EOF from 127.0.98.2:40885 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:45:11.308679 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.302751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:40885)
0927 00:45:11.303037 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.303051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:11.303153 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:11.303416 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:11.303419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:11.303603 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:11.303701 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.303706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.304472 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.304475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.305029 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.305037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.305447 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.305450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.305509 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.306128 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:11.306140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:11.308530 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":68,"threads_started":1}
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:11 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:11.488482 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.482394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:35125)
0927 00:45:11.482758 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.482777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:11.482916 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:11.483195 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:11.483198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:11.483431 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:11.483556 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.483564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.484575 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.484578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.485131 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:11.485138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.485595 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.485598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:11.485661 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.486197 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:11.486208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:11.488314 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
W20260927 00:45:11.488544 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:11.482551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35125 (local address 127.0.98.62:33785)
0927 00:45:11.482723 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:11.482727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:11.482932 (+ 205us) server_negotiation.cc:407] Connection header received
0927 00:45:11.482993 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:11.482997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:11.483065 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:11.483151 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:11.483696 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.484476 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.485252 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:11.485490 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:11.485578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:11.488439 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35125: BlockingRecv error: recv got EOF from 127.0.98.3:35125 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:12.132313 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.127354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:40353)
0927 00:45:12.127635 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.127646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:12.127739 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:12.127979 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:12.127981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:12.128154 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:12.128250 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.128255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.128979 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.128981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.129525 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.129532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.129992 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.129995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.130054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.130514 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:12.130524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:12.132202 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":67,"threads_started":1}
W20260927 00:45:12.132463 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:12.127404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40353 (local address 127.0.98.62:33785)
0927 00:45:12.127635 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:12.127639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:12.127696 (+ 57us) server_negotiation.cc:407] Connection header received
0927 00:45:12.127817 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:12.127819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:12.127860 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:12.127940 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:12.128352 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.128885 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.129616 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.129894 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.129976 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.132328 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40353: BlockingRecv error: recv got EOF from 127.0.98.1:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:12.315445 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.309531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:34081)
0927 00:45:12.309996 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.310015 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:12.310171 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:12.310463 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:12.310466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:12.310665 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:12.310757 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.310762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.311859 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.311862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.312422 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.312431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.312864 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.312867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.312936 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.313432 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:12.313445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:12.315324 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
W20260927 00:45:12.315618 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:12.309704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34081 (local address 127.0.98.62:33785)
0927 00:45:12.309876 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:12.309881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:12.310119 (+ 238us) server_negotiation.cc:407] Connection header received
0927 00:45:12.310267 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:12.310270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:12.310331 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:12.310417 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:12.310931 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.311692 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.312526 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.312737 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.312812 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.315464 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34081: BlockingRecv error: recv got EOF from 127.0.98.2:34081 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:12 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:12.494459 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.489365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:53579)
0927 00:45:12.489622 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.489635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:12.489729 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:12.490008 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:12.490010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:12.490184 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:12.490278 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.490283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.490994 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.490996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.491534 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:12.491541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.491965 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.491967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:12.492026 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.492563 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:12.492574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:12.494342 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
W20260927 00:45:12.494640 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:12.489412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53579 (local address 127.0.98.62:33785)
0927 00:45:12.489555 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:12.489560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:12.489687 (+ 127us) server_negotiation.cc:407] Connection header received
0927 00:45:12.489860 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:12.489863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:12.489909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:12.489975 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:12.490369 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.490907 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.491635 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:12.491876 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:12.491953 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:12.494485 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53579: BlockingRecv error: recv got EOF from 127.0.98.3:53579 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:13.144397 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.133031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:57991)
0927 00:45:13.133333 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.133347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:13.133461 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:13.133733 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:13.133735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:13.133955 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:13.134053 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.134057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.134795 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.134797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.135371 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.135379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.135823 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.135825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.135909 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.142280 (+ 6371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:13.142296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:13.144207 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":122,"threads_started":1}
W20260927 00:45:13.144490 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:13.133077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57991 (local address 127.0.98.62:33785)
0927 00:45:13.133439 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:13.133444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:13.133457 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:13.133569 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:13.133572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:13.133625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:13.133700 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:13.134148 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.134696 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.135493 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.135721 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.135811 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.144347 (+ 8536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57991: BlockingRecv error: recv got EOF from 127.0.98.1:57991 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":69,"threads_started":1}
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:13.322201 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.316224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:50861)
0927 00:45:13.316572 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.316586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:13.316716 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:13.317047 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:13.317050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:13.317256 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:13.317391 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.317398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.318448 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.318451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.319027 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.319035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.319508 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.319511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.319594 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.320185 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:13.320197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:13.322074 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
W20260927 00:45:13.322297 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:13.316336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50861 (local address 127.0.98.62:33785)
0927 00:45:13.316510 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:13.316515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:13.316658 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:45:13.316866 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:13.316868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:13.316914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:13.317012 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:13.317522 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.318335 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.319124 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.319331 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.319449 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.322207 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50861: BlockingRecv error: recv got EOF from 127.0.98.2:50861 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:13 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:13.500676 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.495217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.3:47639)
0927 00:45:13.495522 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.495538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:13.495661 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:13.495902 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:13.495905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:13.496100 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:13.496206 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.496211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.496942 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.496945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.497553 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:13.497560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.498121 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.498124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:13.498190 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.498676 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:13.498688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:13.500549 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":134,"threads_started":1}
W20260927 00:45:13.500849 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:13.495305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47639 (local address 127.0.98.62:33785)
0927 00:45:13.495438 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:13.495441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:13.495623 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:45:13.495741 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:13.495743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:13.495787 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:13.495854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:13.496304 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.496822 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.497659 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:13.498014 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:13.498115 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:13.500684 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47639: BlockingRecv error: recv got EOF from 127.0.98.3:47639 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:45:14 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:14 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:14.150480 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.145233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.1:58153)
0927 00:45:14.145513 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.145530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:14.145604 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:14.145859 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:14.145861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:14.146040 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:14.146134 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.146139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.146827 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.146829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.147383 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.147390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.147828 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.147831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.147894 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.148427 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:14.148439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:14.150307 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260927 00:45:14.150556 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:14.145314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58153 (local address 127.0.98.62:33785)
0927 00:45:14.145548 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:14.145552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:14.145581 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:45:14.145687 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:14.145690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:14.145731 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:14.145827 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:14.146226 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.146736 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.147477 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.147696 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.147770 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.150435 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58153: BlockingRecv error: recv got EOF from 127.0.98.1:58153 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
Sep 27 00:45:14 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:45:14 dist-test-slave-l5sb krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:45:14.328632 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.323154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33785 (local address 127.0.98.2:39709)
0927 00:45:14.323436 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.323450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:14.323550 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:14.323804 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:14.323806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:14.323985 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:14.324078 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.324084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.324822 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.324824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.325404 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.325412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.325909 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.325912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.326018 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.326596 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:14.326614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:14.328506 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33785: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260927 00:45:14.328738 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:14.323320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39709 (local address 127.0.98.62:33785)
0927 00:45:14.323468 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:14.323472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:14.323518 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:45:14.323631 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:14.323633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:14.323674 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:14.323742 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:14.324204 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.324733 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.325502 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.325718 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.325898 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.328629 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39709: BlockingRecv error: recv got EOF from 127.0.98.2:39709 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:45:14.406565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9129
I20260927 00:45:14.412037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9280
I20260927 00:45:14.417413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9414
I20260927 00:45:14.422595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9059
2026-09-27T00:45:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17965 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:14Z Disabled control of system clock
I20260927 00:45:14.452303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:45:14.561551 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:14.561863 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:14.561930 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:14.565806 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:14.565855 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:14.565871 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:14.565887 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:14.565902 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:14.570611 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:14.571604 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:14.572963 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:14.581249 9610 server_base.cc:1034] running on GCE node
W20260927 00:45:14.581332 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:14.581362 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:14.581529 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:14.582110 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:14.583164 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:14.584321 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790469914584293 us; error 42 us; skew 500 ppm
I20260927 00:45:14.586256 9610 webserver.cc:460] Webserver started at http://127.0.98.62:44273/ using document root <none> and password file <none>
I20260927 00:45:14.586824 9610 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:14.586931 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:14.587127 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:14.589054 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "b21880f3fff646b99a0a68c2a921d353"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "331dfed011040d79d296454b95d2d8ce"
server_key_iv: "34f321329b9fb4de32f1ae1c97a1fcb2"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.589555 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "b21880f3fff646b99a0a68c2a921d353"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "331dfed011040d79d296454b95d2d8ce"
server_key_iv: "34f321329b9fb4de32f1ae1c97a1fcb2"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.592993 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:14.595214 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:14.596411 9610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:14.596541 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "b21880f3fff646b99a0a68c2a921d353"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "331dfed011040d79d296454b95d2d8ce"
server_key_iv: "34f321329b9fb4de32f1ae1c97a1fcb2"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.596643 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:14.617004 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:14.617677 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:14.617882 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:14.624936 9610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34843
I20260927 00:45:14.624946 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34843 every 8 connection(s)
I20260927 00:45:14.625836 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:14.627633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9610
I20260927 00:45:14.627764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:14.628078 392 external_mini_cluster.cc:1428] Setting key 1937d4fa3b2e2753f8bc6f61bff8f2e4
I20260927 00:45:14.628964 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:14.634402 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353: Bootstrap starting.
I20260927 00:45:14.636669 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:14.636716 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.629403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.0.98.62:34843)
0927 00:45:14.630035 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:14.630050 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:14.630087 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:45:14.630724 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:14.630739 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:14.631101 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:14.631425 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:14.631810 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.632725 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.633602 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.634119 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.634282 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.635141 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:14.635146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:14.635158 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:14.635515 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:14.635566 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:14.635584 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:14.635686 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:14.635809 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 00:45:14.636044 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260927 00:45:14.637554 9677 log.cc:824] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:14.639426 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353: No bootstrap required, opened a new log
I20260927 00:45:14.642871 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } }
I20260927 00:45:14.643067 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:14.643121 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21880f3fff646b99a0a68c2a921d353, State: Initialized, Role: FOLLOWER
I20260927 00:45:14.643509 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } }
I20260927 00:45:14.643630 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:14.643672 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:14.643743 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:14.644562 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } }
I20260927 00:45:14.644842 9677 leader_election.cc:302] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21880f3fff646b99a0a68c2a921d353; no voters:
I20260927 00:45:14.645079 9677 leader_election.cc:288] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:14.645215 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:14.645484 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [term 1 LEADER]: Becoming Leader. State: Replica: b21880f3fff646b99a0a68c2a921d353, State: Running, Role: LEADER
I20260927 00:45:14.645871 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } }
I20260927 00:45:14.646341 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:14.647491 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21880f3fff646b99a0a68c2a921d353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } } }
I20260927 00:45:14.647630 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:14.647980 9691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:14.648361 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21880f3fff646b99a0a68c2a921d353. Latest consensus state: current_term: 1 leader_uuid: "b21880f3fff646b99a0a68c2a921d353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21880f3fff646b99a0a68c2a921d353" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34843 } } }
I20260927 00:45:14.648464 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:14.651229 9691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:14.656560 9691 catalog_manager.cc:1347] Generated new cluster ID: 2cffa1d21e684f718f218c0da2f908fb
I20260927 00:45:14.656620 9691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:14.676786 9691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:14.677699 9691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:14.686821 9691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21880f3fff646b99a0a68c2a921d353: Generated new TSK 0
I20260927 00:45:14.687584 9691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:14.694564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:45:14.809486 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:14.809753 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:14.809837 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:14.813832 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:14.813912 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:14.813998 9701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:14.818733 9701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:14.819737 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:14.821058 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:14.830561 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:14.830612 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:14.830561 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:14.830943 9701 server_base.cc:1034] running on GCE node
I20260927 00:45:14.831261 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:14.831779 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:14.832937 9701 hybrid_clock.cc:648] HybridClock initialized: now 1790469914832917 us; error 31 us; skew 500 ppm
I20260927 00:45:14.834836 9701 webserver.cc:460] Webserver started at http://127.0.98.1:43159/ using document root <none> and password file <none>
I20260927 00:45:14.835422 9701 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:14.835487 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:14.835733 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:14.837909 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "eda0de66057e465783d2c25f3e3402e2"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "d27c26fa85e6e39e917eb8e87ebcb908"
server_key_iv: "8d2bad7a609cd1d7ca4cba24989e7f04"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.838491 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eda0de66057e465783d2c25f3e3402e2"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "d27c26fa85e6e39e917eb8e87ebcb908"
server_key_iv: "8d2bad7a609cd1d7ca4cba24989e7f04"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.842267 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:14.844637 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:14.845906 9701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:14.846021 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "eda0de66057e465783d2c25f3e3402e2"
format_stamp: "Formatted at 2026-09-27 00:45:14 on dist-test-slave-l5sb"
server_key: "d27c26fa85e6e39e917eb8e87ebcb908"
server_key_iv: "8d2bad7a609cd1d7ca4cba24989e7f04"
server_key_version: "encryptionkey@0"
I20260927 00:45:14.846149 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:14.858682 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:14.859361 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:14.859550 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:14.860126 9701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:14.861042 9701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:14.861112 9701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:14.861172 9701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:14.861219 9701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:14.869668 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33631
I20260927 00:45:14.869729 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33631 every 8 connection(s)
I20260927 00:45:14.870613 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:14.871454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9701
I20260927 00:45:14.871568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:14.871876 392 external_mini_cluster.cc:1428] Setting key f8560cd0afccc9b4bb5492c254969322
I20260927 00:45:14.874125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:45:14.881863 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.872620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49641 (local address 127.0.98.62:34843)
0927 00:45:14.872779 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:14.872784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:14.874110 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 00:45:14.876244 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:14.876250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:14.876312 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:14.876419 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:14.877371 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.878306 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.879539 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.879957 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.880091 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.881216 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:14.881220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:14.881224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:14.881242 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:14.881269 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:14.881283 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:14.881378 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:14.881529 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:45:14.881624 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:45:14.882679 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.873084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34843 (local address 127.0.98.1:49641)
0927 00:45:14.873938 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.874000 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:14.876031 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:14.876575 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:14.876582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:14.876668 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:14.877210 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.877223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.878441 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.878444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.879408 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:14.879417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:14.880108 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:14.880110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:14.880259 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:14.880935 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:14.880937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:14.880980 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:14.881048 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:14.881383 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:14.881390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:14.881394 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:45:14.881495 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:45:14.881726 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260927 00:45:14.884017 9829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34843
I20260927 00:45:14.884289 9829 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:14.884974 9829 heartbeater.cc:511] Master 127.0.98.62:34843 requested a full tablet report, sending...
I20260927 00:45:14.886528 9641 ts_manager.cc:194] Registered new tserver with Master: eda0de66057e465783d2c25f3e3402e2 (127.0.98.1:33631)
I20260927 00:45:14.887743 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49641
W20260927 00:45:14.985522 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:14.985805 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:14.985844 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:14.989976 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:14.990032 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:14.990118 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:14.994871 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:14.995977 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:14.997331 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:15.006613 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.006795 9832 server_base.cc:1034] running on GCE node
W20260927 00:45:15.006619 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.006680 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.007262 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:15.007962 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:15.009119 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790469915009102 us; error 27 us; skew 500 ppm
I20260927 00:45:15.010960 9832 webserver.cc:460] Webserver started at http://127.0.98.2:35937/ using document root <none> and password file <none>
I20260927 00:45:15.011528 9832 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:15.011588 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:15.011825 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:15.013860 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "4913c76997944dc688bf0de1133af66a"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "0f856f534384ae32d24dddfe5b2304e1"
server_key_iv: "f008c03105d25c89affc2162594f70f6"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.014402 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4913c76997944dc688bf0de1133af66a"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "0f856f534384ae32d24dddfe5b2304e1"
server_key_iv: "f008c03105d25c89affc2162594f70f6"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.018112 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:45:15.020794 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.022028 9832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:15.022147 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "4913c76997944dc688bf0de1133af66a"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "0f856f534384ae32d24dddfe5b2304e1"
server_key_iv: "f008c03105d25c89affc2162594f70f6"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.022254 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:15.034384 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:15.035008 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:15.035184 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:15.035734 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:15.036610 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:15.036674 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.036739 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:15.036785 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.045728 9832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36897
I20260927 00:45:15.045753 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36897 every 8 connection(s)
I20260927 00:45:15.046763 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:15.050688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9832
I20260927 00:45:15.050808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:15.051193 392 external_mini_cluster.cc:1428] Setting key 25af457969ae8418f867f7d471092ecb
I20260927 00:45:15.053215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:45:15.057240 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.048600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47797 (local address 127.0.98.62:34843)
0927 00:45:15.048750 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.048755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.049905 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 00:45:15.051743 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.051748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.051820 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.051891 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.052949 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.053537 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.055131 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.055509 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.055639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:15.056661 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:15.056668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:15.056674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:15.056689 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:15.056715 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:15.056728 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:15.056816 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:15.056957 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:45:15.057032 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:45:15.058020 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.049094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34843 (local address 127.0.98.2:47797)
0927 00:45:15.049725 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.049765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:15.051509 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:15.052080 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:15.052087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:15.052169 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:15.052811 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:15.052831 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.053657 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.053664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:15.055014 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:15.055024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.055648 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.055650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:15.055788 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:15.056407 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:15.056409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:15.056448 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:15.056508 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:15.056817 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:15.056824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:15.056828 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:45:15.056927 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:45:15.057138 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260927 00:45:15.059152 9960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34843
I20260927 00:45:15.059367 9960 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:15.059844 9960 heartbeater.cc:511] Master 127.0.98.62:34843 requested a full tablet report, sending...
I20260927 00:45:15.060786 9641 ts_manager.cc:194] Registered new tserver with Master: 4913c76997944dc688bf0de1133af66a (127.0.98.2:36897)
I20260927 00:45:15.061367 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47797
W20260927 00:45:15.166071 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:15.166322 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:15.166364 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:15.170325 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:15.170379 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:15.170464 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:15.175051 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:15.176031 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:15.177330 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:15.186483 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.186584 9963 server_base.cc:1034] running on GCE node
W20260927 00:45:15.186493 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.186453 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.187252 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:15.187815 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:15.189003 9963 hybrid_clock.cc:648] HybridClock initialized: now 1790469915188984 us; error 29 us; skew 500 ppm
I20260927 00:45:15.190919 9963 webserver.cc:460] Webserver started at http://127.0.98.3:37295/ using document root <none> and password file <none>
I20260927 00:45:15.191455 9963 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:15.191512 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:15.191673 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:15.193616 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "b7a8e0589129415b9264689e03fdd341"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "c5680d57423379c8cacf0bdbf3026317"
server_key_iv: "ec1139a5f6e8231725336f3e3add5a74"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.194193 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b7a8e0589129415b9264689e03fdd341"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "c5680d57423379c8cacf0bdbf3026317"
server_key_iv: "ec1139a5f6e8231725336f3e3add5a74"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.197603 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:45:15.200004 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.201201 9963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:15.201367 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "b7a8e0589129415b9264689e03fdd341"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "c5680d57423379c8cacf0bdbf3026317"
server_key_iv: "ec1139a5f6e8231725336f3e3add5a74"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.201463 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:15.214936 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:15.215580 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:15.215801 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:15.216359 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:15.217254 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:15.217320 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.217396 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:15.217428 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.226405 9963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35443
I20260927 00:45:15.226467 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35443 every 8 connection(s)
I20260927 00:45:15.227360 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:15.229477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 9963
I20260927 00:45:15.229599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:15.229943 392 external_mini_cluster.cc:1428] Setting key ef42277d681953e2e0e521f1d928493d
I20260927 00:45:15.236872 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.229152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56155 (local address 127.0.98.62:34843)
0927 00:45:15.229311 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.229316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.230577 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 00:45:15.232226 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.232228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.232270 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.232333 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.233186 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.233700 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.234892 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.235115 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.235207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:15.236284 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:15.236287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:15.236290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:15.236302 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:15.236339 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:15.236367 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:15.236446 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:15.236617 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:45:15.236710 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:15.237942 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.229611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34843 (local address 127.0.98.3:56155)
0927 00:45:15.230417 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.230467 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:15.232052 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:15.232465 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:15.232472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:15.232555 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:15.233066 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:15.233078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.233883 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.233886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:15.234761 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:15.234770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:15.235265 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:15.235268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:15.235415 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:15.236037 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:15.236039 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:15.236081 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:15.236145 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:15.236453 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:15.236463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:15.236472 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:45:15.236609 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:45:15.236943 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":165,"threads_started":1}
I20260927 00:45:15.238991 10091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34843
I20260927 00:45:15.239230 10091 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:15.239760 10091 heartbeater.cc:511] Master 127.0.98.62:34843 requested a full tablet report, sending...
I20260927 00:45:15.240732 9641 ts_manager.cc:194] Registered new tserver with Master: b7a8e0589129415b9264689e03fdd341 (127.0.98.3:35443)
I20260927 00:45:15.241334 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56155
I20260927 00:45:15.243680 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:45:15.246613 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:15.245525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.0.98.62:34843)
0927 00:45:15.245642 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.245646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.245960 (+ 314us) server_negotiation.cc:407] Connection header received
0927 00:45:15.246117 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.246120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.246163 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.246246 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.246498 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41550: BlockingRecv error: recv got EOF from 127.0.0.1:41550 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:45:15.246731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9701
I20260927 00:45:15.252894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9832
I20260927 00:45:15.258500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9963
I20260927 00:45:15.263960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 9610
2026-09-27T00:45:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:15Z Disabled control of system clock
I20260927 00:45:15.292948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38159
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:45:15.412567 10102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:15.412859 10102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:15.412902 10102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:15.417531 10102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:15.417593 10102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:15.417610 10102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:15.417629 10102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:15.417680 10102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:15.423022 10102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:15.424161 10102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:15.425630 10102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:15.434150 10110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.434150 10108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.434173 10107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.434387 10102 server_base.cc:1034] running on GCE node
I20260927 00:45:15.434899 10102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:15.436023 10102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:15.437207 10102 hybrid_clock.cc:648] HybridClock initialized: now 1790469915437193 us; error 28 us; skew 500 ppm
I20260927 00:45:15.439126 10102 webserver.cc:460] Webserver started at http://127.0.98.62:39357/ using document root <none> and password file <none>
I20260927 00:45:15.439682 10102 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:15.439767 10102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:15.440099 10102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:15.442281 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "483bffc567154ba0bfc6005831278f1d"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "227325b20c8086193d56baa02833bda5"
server_key_iv: "507d1fd3629169462dd292ad4b0e0056"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.442843 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "483bffc567154ba0bfc6005831278f1d"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "227325b20c8086193d56baa02833bda5"
server_key_iv: "507d1fd3629169462dd292ad4b0e0056"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.446659 10102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:45:15.449180 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.450461 10102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:15.450611 10102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "483bffc567154ba0bfc6005831278f1d"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "227325b20c8086193d56baa02833bda5"
server_key_iv: "507d1fd3629169462dd292ad4b0e0056"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.450727 10102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:15.455291 10102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:15.455892 10102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:15.456411 10102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:15.463369 10102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38159
I20260927 00:45:15.463384 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38159 every 8 connection(s)
I20260927 00:45:15.464461 10102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:15.467784 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:15.468741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10102
I20260927 00:45:15.468844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:15.469156 392 external_mini_cluster.cc:1428] Setting key 08590f9826aaac33177c908a0219978f
I20260927 00:45:15.474121 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d: Bootstrap starting.
I20260927 00:45:15.474877 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.470647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.0.98.62:38159)
0927 00:45:15.471321 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.471337 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.471374 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:45:15.472011 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.472027 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.472381 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.472665 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.473516 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:15.473521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:15.473532 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:15.473542 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:15.473595 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:15.473611 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:15.473720 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:15.473888 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:45:15.474149 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":184,"threads_started":1}
I20260927 00:45:15.476317 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:15.477097 10169 log.cc:824] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:15.478906 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d: No bootstrap required, opened a new log
I20260927 00:45:15.482540 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } }
I20260927 00:45:15.482787 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:15.482864 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483bffc567154ba0bfc6005831278f1d, State: Initialized, Role: FOLLOWER
I20260927 00:45:15.483297 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } }
I20260927 00:45:15.483436 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:15.483515 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:15.483611 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:15.484570 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } }
I20260927 00:45:15.484901 10169 leader_election.cc:302] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 483bffc567154ba0bfc6005831278f1d; no voters:
I20260927 00:45:15.485239 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:15.485332 10174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:15.485581 10174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [term 1 LEADER]: Becoming Leader. State: Replica: 483bffc567154ba0bfc6005831278f1d, State: Running, Role: LEADER
I20260927 00:45:15.485908 10174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } }
I20260927 00:45:15.486501 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:15.487463 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 483bffc567154ba0bfc6005831278f1d. Latest consensus state: current_term: 1 leader_uuid: "483bffc567154ba0bfc6005831278f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } } }
I20260927 00:45:15.487596 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:15.488610 10183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:15.488679 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "483bffc567154ba0bfc6005831278f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483bffc567154ba0bfc6005831278f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38159 } } }
I20260927 00:45:15.489156 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:15.491359 10183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:15.496984 10183 catalog_manager.cc:1347] Generated new cluster ID: 444e6f466d7b428d903071f75d727355
I20260927 00:45:15.497079 10183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:15.508488 10183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:15.508834 10183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:15.518482 10183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 483bffc567154ba0bfc6005831278f1d: Generated new TSK 0
I20260927 00:45:15.519382 10183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:15.532135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:45:15.640606 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:15.640842 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:15.640919 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:15.644863 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:15.644918 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:15.644997 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:15.649617 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:15.650627 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:15.651937 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:15.661156 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.661337 10193 server_base.cc:1034] running on GCE node
W20260927 00:45:15.661172 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.661168 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.661896 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:15.662477 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:15.663628 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790469915663613 us; error 26 us; skew 500 ppm
I20260927 00:45:15.665421 10193 webserver.cc:460] Webserver started at http://127.0.98.1:39625/ using document root <none> and password file <none>
I20260927 00:45:15.666024 10193 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:15.666087 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:15.666319 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:15.668347 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "d39595ea70c04f62ac84fbaa072af346"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "68a45163eac6da48fb103c5c92d0b679"
server_key_iv: "73c3f1adbb6e2a6ae476d86427310236"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.668890 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d39595ea70c04f62ac84fbaa072af346"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "68a45163eac6da48fb103c5c92d0b679"
server_key_iv: "73c3f1adbb6e2a6ae476d86427310236"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.672382 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:15.674655 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.675787 10193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:15.675916 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "d39595ea70c04f62ac84fbaa072af346"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "68a45163eac6da48fb103c5c92d0b679"
server_key_iv: "73c3f1adbb6e2a6ae476d86427310236"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.676018 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:15.680455 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:15.681007 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:15.681172 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:15.681669 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:15.682572 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:15.682637 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.682701 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:15.682746 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.691548 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43981
I20260927 00:45:15.691570 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43981 every 8 connection(s)
I20260927 00:45:15.692404 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:15.698014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10193
I20260927 00:45:15.698130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:15.698151 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.694090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33159 (local address 127.0.98.62:38159)
0927 00:45:15.694214 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.694217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.695208 (+ 991us) server_negotiation.cc:407] Connection header received
0927 00:45:15.696505 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.696508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.696553 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.696654 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.697673 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:15.697676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:15.697679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:15.697691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:15.697717 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:15.697726 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:15.697845 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:15.697979 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:45:15.698022 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:45:15.698482 392 external_mini_cluster.cc:1428] Setting key 428e7b49c0ecf062d13a1676b8fa9c53
I20260927 00:45:15.699048 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.694449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38159 (local address 127.0.98.1:33159)
0927 00:45:15.695071 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.695109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:15.696285 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:15.696824 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:15.696831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:15.696916 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:15.697439 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:15.697440 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:15.697476 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:15.697538 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:15.697824 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:15.697832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:15.697838 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:45:15.698008 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 00:45:15.698295 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260927 00:45:15.700197 10321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38159
I20260927 00:45:15.700382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:45:15.700418 10321 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:15.700793 10321 heartbeater.cc:511] Master 127.0.98.62:38159 requested a full tablet report, sending...
I20260927 00:45:15.702181 10132 ts_manager.cc:194] Registered new tserver with Master: d39595ea70c04f62ac84fbaa072af346 (127.0.98.1:43981)
W20260927 00:45:15.808773 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:15.809044 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:15.809113 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:15.813158 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:15.813246 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:15.813330 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:15.818199 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:15.819237 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:15.820554 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:15.829468 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.829468 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:15.829533 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:15.829841 10324 server_base.cc:1034] running on GCE node
I20260927 00:45:15.830232 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:15.830853 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:15.832027 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790469915832014 us; error 25 us; skew 500 ppm
I20260927 00:45:15.833745 10324 webserver.cc:460] Webserver started at http://127.0.98.2:37735/ using document root <none> and password file <none>
I20260927 00:45:15.834352 10324 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:15.834438 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:15.834654 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:15.836586 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "80a0d966e0554bafa68b2f956bf8f8aa"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "f8dc1e7fe7e14a396b4286d5de5cffb8"
server_key_iv: "86f7438c1b65217b8d681a8003842092"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.837131 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80a0d966e0554bafa68b2f956bf8f8aa"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "f8dc1e7fe7e14a396b4286d5de5cffb8"
server_key_iv: "86f7438c1b65217b8d681a8003842092"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.840631 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:15.842931 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.844071 10324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:15.844194 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "80a0d966e0554bafa68b2f956bf8f8aa"
format_stamp: "Formatted at 2026-09-27 00:45:15 on dist-test-slave-l5sb"
server_key: "f8dc1e7fe7e14a396b4286d5de5cffb8"
server_key_iv: "86f7438c1b65217b8d681a8003842092"
server_key_version: "encryptionkey@0"
I20260927 00:45:15.844298 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:15.848711 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:15.849292 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:15.849468 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:15.850034 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:15.850916 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:15.850982 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.851045 10324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:15.851089 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:15.860028 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41745
I20260927 00:45:15.860046 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41745 every 8 connection(s)
I20260927 00:45:15.860960 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:15.866039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10324
I20260927 00:45:15.866169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:15.866509 392 external_mini_cluster.cc:1428] Setting key d2f63455cdcb60134168acfff476d592
I20260927 00:45:15.866930 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.862761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41987 (local address 127.0.98.62:38159)
0927 00:45:15.862901 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:15.862905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:15.863939 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 00:45:15.865150 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:15.865152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:15.865187 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:15.865248 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:15.866423 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:15.866427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:15.866429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:15.866444 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:15.866469 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:15.866479 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:15.866592 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:15.866754 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:45:15.866799 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:15.867861 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.863163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38159 (local address 127.0.98.2:41987)
0927 00:45:15.863791 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.863829 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:15.864959 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:15.865372 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:15.865378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:15.865461 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:15.866104 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:15.866110 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:15.866165 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:15.866260 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:15.866589 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:15.866599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:15.866606 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:45:15.866748 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 00:45:15.866935 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260927 00:45:15.868870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:45:15.869094 10452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38159
I20260927 00:45:15.869318 10452 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:15.869695 10452 heartbeater.cc:511] Master 127.0.98.62:38159 requested a full tablet report, sending...
I20260927 00:45:15.870692 10132 ts_manager.cc:194] Registered new tserver with Master: 80a0d966e0554bafa68b2f956bf8f8aa (127.0.98.2:41745)
W20260927 00:45:15.981389 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:15.981664 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:15.981706 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:15.985966 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:15.986027 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:15.986084 10455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:15.991488 10455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:15.992599 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:15.994050 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:16.003391 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:16.003394 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:16.003494 10455 server_base.cc:1034] running on GCE node
W20260927 00:45:16.003404 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:16.003979 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:16.004580 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:16.005825 10455 hybrid_clock.cc:648] HybridClock initialized: now 1790469916005793 us; error 49 us; skew 500 ppm
I20260927 00:45:16.007628 10455 webserver.cc:460] Webserver started at http://127.0.98.3:46779/ using document root <none> and password file <none>
I20260927 00:45:16.008183 10455 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:16.008263 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:16.008464 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:16.010560 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "5db87d4490ca4265a8cb97577e6d6e2d"
format_stamp: "Formatted at 2026-09-27 00:45:16 on dist-test-slave-l5sb"
server_key: "c8647e7a298c1642240ac478956d9b48"
server_key_iv: "6fe29f117402d9e4ce1d9c9697413870"
server_key_version: "encryptionkey@0"
I20260927 00:45:16.011114 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5db87d4490ca4265a8cb97577e6d6e2d"
format_stamp: "Formatted at 2026-09-27 00:45:16 on dist-test-slave-l5sb"
server_key: "c8647e7a298c1642240ac478956d9b48"
server_key_iv: "6fe29f117402d9e4ce1d9c9697413870"
server_key_version: "encryptionkey@0"
I20260927 00:45:16.014688 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:16.017015 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:16.018409 10455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:16.018522 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "5db87d4490ca4265a8cb97577e6d6e2d"
format_stamp: "Formatted at 2026-09-27 00:45:16 on dist-test-slave-l5sb"
server_key: "c8647e7a298c1642240ac478956d9b48"
server_key_iv: "6fe29f117402d9e4ce1d9c9697413870"
server_key_version: "encryptionkey@0"
I20260927 00:45:16.018644 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:16.023440 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:16.023983 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:16.024116 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:16.024578 10455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:16.025449 10455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:16.025493 10455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:16.025528 10455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:16.025549 10455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:16.033691 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39561
I20260927 00:45:16.033751 10582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39561 every 8 connection(s)
I20260927 00:45:16.034601 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:16.035082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10455
I20260927 00:45:16.035194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:16.035513 392 external_mini_cluster.cc:1428] Setting key e24e545003a63c680e20ee52bf47b162
I20260927 00:45:16.040534 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:16.036392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35439 (local address 127.0.98.62:38159)
0927 00:45:16.036492 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:16.036495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:16.037814 (+ 1319us) server_negotiation.cc:407] Connection header received
0927 00:45:16.039091 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:16.039093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:16.039130 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:16.039202 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:16.040179 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:16.040182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:16.040184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:16.040194 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:16.040213 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:16.040221 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:16.040301 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:16.040406 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:45:16.040440 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:45:16.041322 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:16.036916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38159 (local address 127.0.98.3:35439)
0927 00:45:16.037661 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:16.037710 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:16.038901 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:16.039323 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:16.039329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:16.039421 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:16.039939 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:16.039940 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:16.039978 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:16.040043 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:16.040291 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:16.040295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:16.040298 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:45:16.040425 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:45:16.040602 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260927 00:45:16.042343 10583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38159
I20260927 00:45:16.042529 10583 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:16.042860 10583 heartbeater.cc:511] Master 127.0.98.62:38159 requested a full tablet report, sending...
I20260927 00:45:16.043757 10132 ts_manager.cc:194] Registered new tserver with Master: 5db87d4490ca4265a8cb97577e6d6e2d (127.0.98.3:39561)
I20260927 00:45:16.049741 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:45:16.052533 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:16.051641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.98.62:38159)
0927 00:45:16.051780 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:16.051784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:16.051976 (+ 192us) server_negotiation.cc:407] Connection header received
0927 00:45:16.052088 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:16.052090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:16.052120 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:16.052183 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:16.052412 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52058: BlockingRecv error: recv got EOF from 127.0.0.1:52058 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:45:16.052678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10193
I20260927 00:45:16.058370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10324
I20260927 00:45:16.064172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10455
I20260927 00:45:16.069285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10102
2026-09-27T00:45:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (805 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:16 dist-test-slave-l5sb krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:16 dist-test-slave-l5sb krb5kdc[10593](info): set up 2 sockets
Sep 27 00:45:16 dist-test-slave-l5sb krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260927 00:45:17.162933 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:17.286049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:17.394008 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:17.394299 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:17.394376 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:17.398401 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:17.398481 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:17.398519 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:17.398571 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:17.398612 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:17.403947 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:17.405038 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:17.406132 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:17.414680 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.414810 10606 server_base.cc:1034] running on GCE node
W20260927 00:45:17.414845 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.414753 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.415406 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:17.416509 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:17.417706 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790469917417693 us; error 29 us; skew 500 ppm
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:17.420548 10606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:17.421470 10606 webserver.cc:460] Webserver started at http://127.0.98.62:39693/ using document root <none> and password file <none>
I20260927 00:45:17.422040 10606 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:17.422101 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:17.422266 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:17.424211 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "10048b011b93435f9ab916da11420f21"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "c0860dbbbfda71c65ac20eb42d912de9"
server_key_iv: "6bd6bcbbcdd44aaf97ed78702a1f8270"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.424713 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "10048b011b93435f9ab916da11420f21"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "c0860dbbbfda71c65ac20eb42d912de9"
server_key_iv: "6bd6bcbbcdd44aaf97ed78702a1f8270"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.428305 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:17.430647 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.431926 10606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:17.432061 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "10048b011b93435f9ab916da11420f21"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "c0860dbbbfda71c65ac20eb42d912de9"
server_key_iv: "6bd6bcbbcdd44aaf97ed78702a1f8270"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.432173 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:17.450116 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:17.453107 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:17.453305 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:17.460258 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37047
I20260927 00:45:17.460558 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37047 every 8 connection(s)
I20260927 00:45:17.461267 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:17.461637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10606
I20260927 00:45:17.461795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:17.462152 392 external_mini_cluster.cc:1428] Setting key eaac279195f05bec70e8249e07bb07c3
I20260927 00:45:17.464579 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:17.471244 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: Bootstrap starting.
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:17.473831 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:17.474753 10674 log.cc:824] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:17.476774 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.463548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.98.62:37047)
0927 00:45:17.463977 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:17.463994 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:17.464037 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:45:17.464882 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:17.464905 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:17.465304 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:17.465654 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:17.466370 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.467857 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.468658 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.469351 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.469501 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.472132 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:17.472137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:17.472149 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:17.472176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:17.473759 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.474278 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.474283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.474287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.474391 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.474666 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.474668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.474669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.475094 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:17.475229 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:17.475586 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 00:45:17.475900 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":105,"threads_started":1}
I20260927 00:45:17.476862 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: No bootstrap required, opened a new log
I20260927 00:45:17.480505 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 00:45:17.480712 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:17.480746 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10048b011b93435f9ab916da11420f21, State: Initialized, Role: FOLLOWER
I20260927 00:45:17.481092 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 00:45:17.481195 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:17.481238 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:17.481304 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:17.482199 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 00:45:17.482489 10674 leader_election.cc:302] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10048b011b93435f9ab916da11420f21; no voters:
I20260927 00:45:17.482748 10674 leader_election.cc:288] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:17.482906 10679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:17.483136 10679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [term 1 LEADER]: Becoming Leader. State: Replica: 10048b011b93435f9ab916da11420f21, State: Running, Role: LEADER
I20260927 00:45:17.483455 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 00:45:17.484038 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:17.485100 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10048b011b93435f9ab916da11420f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260927 00:45:17.485226 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:17.485507 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10048b011b93435f9ab916da11420f21. Latest consensus state: current_term: 1 leader_uuid: "10048b011b93435f9ab916da11420f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10048b011b93435f9ab916da11420f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260927 00:45:17.485579 10688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:17.485594 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:17.488296 10688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:17.494138 10688 catalog_manager.cc:1347] Generated new cluster ID: 94c24930db0c4c0196140e81bd1cdae0
I20260927 00:45:17.494210 10688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:17.505374 10688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:17.506521 10688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:17.512013 10688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: Generated new TSK 0
I20260927 00:45:17.512702 10688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:45:17.552973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:17.669631 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:17.669925 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:17.669971 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:17.673882 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:17.673935 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:17.673990 10700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:17.679215 10700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:17.680198 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:17.681171 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:17.690950 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.690949 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.690968 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.691035 10700 server_base.cc:1034] running on GCE node
I20260927 00:45:17.691594 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:17.692296 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:17.693488 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790469917693472 us; error 28 us; skew 500 ppm
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:17.696367 10700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:45:17.697175 10700 webserver.cc:460] Webserver started at http://127.0.98.1:37195/ using document root <none> and password file <none>
I20260927 00:45:17.697695 10700 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:17.697762 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:17.697993 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:17.699975 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad8b7e208bcc4169940149f725f65e41"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "fe8bfd189166a6d9f617e223097d422e"
server_key_iv: "b349cb31941e8f7d73bcf71a45660f61"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.700459 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad8b7e208bcc4169940149f725f65e41"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "fe8bfd189166a6d9f617e223097d422e"
server_key_iv: "b349cb31941e8f7d73bcf71a45660f61"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.704169 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:45:17.706943 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.708217 10700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:17.708329 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "ad8b7e208bcc4169940149f725f65e41"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "fe8bfd189166a6d9f617e223097d422e"
server_key_iv: "b349cb31941e8f7d73bcf71a45660f61"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.708413 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:17.735973 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:17.739101 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:17.739303 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:17.739934 10700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:17.740804 10700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:17.740851 10700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.740923 10700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:17.740953 10700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.750161 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33857
I20260927 00:45:17.750221 10828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33857 every 8 connection(s)
I20260927 00:45:17.751143 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:17.759948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10700
I20260927 00:45:17.760095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:17.760439 392 external_mini_cluster.cc:1428] Setting key d4a1d732bb4c8cf3dc3dc80923576804
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:17.764732 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.752897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40091 (local address 127.0.98.62:37047)
0927 00:45:17.753034 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:17.753038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:17.753932 (+ 894us) server_negotiation.cc:407] Connection header received
0927 00:45:17.755215 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:17.755219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:17.755272 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:17.755357 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:17.756500 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.757032 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.758213 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.758444 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.758550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.761217 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:17.761220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:17.761223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:17.761256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:17.762766 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.763263 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.763269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.763271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.763347 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.763669 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.763674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.763676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.763852 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:17.763953 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:17.764429 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 00:45:17.764560 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:45:17.765523 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.753296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.1:40091)
0927 00:45:17.753727 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:17.753820 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:17.755011 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:17.755527 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:17.755534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:17.755891 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:17.756359 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.756372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.757155 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.757163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.758099 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.758108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.758586 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.758590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.758759 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.759345 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:17.759360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:17.761052 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:17.762895 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.762903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.762919 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.763167 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.763444 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.763447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.763449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.763581 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.763950 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:17.763957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:17.764209 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 00:45:17.764392 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:45:17.764707 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
I20260927 00:45:17.766746 10829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 00:45:17.766988 10829 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:17.767442 10829 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
I20260927 00:45:17.769021 10638 ts_manager.cc:194] Registered new tserver with Master: ad8b7e208bcc4169940149f725f65e41 (127.0.98.1:33857)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:45:17.770320 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:45:17.788420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:17.897318 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:17.897598 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:17.897670 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:17.901714 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:17.901793 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:17.901876 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:17.907092 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:17.908169 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:17.909186 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:17.918606 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.918606 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.918624 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.918980 10834 server_base.cc:1034] running on GCE node
I20260927 00:45:17.919445 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:17.920195 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:17.921370 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790469917921336 us; error 48 us; skew 500 ppm
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:17.924235 10834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:45:17.925050 10834 webserver.cc:460] Webserver started at http://127.0.98.2:40591/ using document root <none> and password file <none>
I20260927 00:45:17.925590 10834 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:17.925671 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:17.925935 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:17.927886 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "48c0c438538a4cb78f03010b39738933"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "6e7c24e057574e135e2617f596530a75"
server_key_iv: "804277a4d054845c133e35bd61b39012"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.928455 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48c0c438538a4cb78f03010b39738933"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "6e7c24e057574e135e2617f596530a75"
server_key_iv: "804277a4d054845c133e35bd61b39012"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.931998 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:17.934389 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.935585 10834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:45:17.935717 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "48c0c438538a4cb78f03010b39738933"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-l5sb"
server_key: "6e7c24e057574e135e2617f596530a75"
server_key_iv: "804277a4d054845c133e35bd61b39012"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.935827 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:17.953114 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:17.956331 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:17.956540 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:17.957132 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:17.958011 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:17.958079 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.958151 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:17.958189 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.966976 10834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45159
I20260927 00:45:17.967000 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45159 every 8 connection(s)
I20260927 00:45:17.967900 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:17.974522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10834
I20260927 00:45:17.974632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:17.974934 392 external_mini_cluster.cc:1428] Setting key 44560eca7d7d6439740c3ddfbc79205f
Sep 27 00:45:17 dist-test-slave-l5sb krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:17.984365 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.969865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41011 (local address 127.0.98.62:37047)
0927 00:45:17.970001 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:17.970005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:17.970940 (+ 935us) server_negotiation.cc:407] Connection header received
0927 00:45:17.972870 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:17.972874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:17.972938 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:17.973047 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:17.974429 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.975074 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.976684 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.976916 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.977003 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.980215 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:17.980222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:17.980228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:17.980261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:17.981932 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.982644 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.982646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.982648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.982720 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.983152 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.983154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.983155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.983330 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:17.983403 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:17.984129 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 00:45:17.984239 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:17.985724 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.970254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.2:41011)
0927 00:45:17.970767 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:17.970814 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:17.972638 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:17.973281 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:17.973289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:17.973686 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:17.974260 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.974282 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.975201 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.975206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.976566 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.976579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.977092 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.977099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.977379 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.978121 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:17.978150 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:17.980043 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:17.982045 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.982054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.982076 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.982525 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.982814 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.982819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.982824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.983042 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.983441 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:17.983448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:17.983812 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 00:45:17.984052 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 00:45:17.984464 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:45:17.986925 10963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 00:45:17.987197 10963 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:17.987742 10963 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:45:17.988987 10638 ts_manager.cc:194] Registered new tserver with Master: 48c0c438538a4cb78f03010b39738933 (127.0.98.2:45159)
I20260927 00:45:17.989617 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:45:18.005664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:18.114181 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:18.114494 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:18.114575 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:18.118516 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:18.118592 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:18.118686 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:18.123950 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:45:18.124960 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:18.125995 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:18.135207 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.135281 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.135304 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.135447 10968 server_base.cc:1034] running on GCE node
I20260927 00:45:18.135973 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:18.136636 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:18.137835 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790469918137808 us; error 34 us; skew 500 ppm
Sep 27 00:45:18 dist-test-slave-l5sb krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:18.140558 10968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:45:18.141420 10968 webserver.cc:460] Webserver started at http://127.0.98.3:38883/ using document root <none> and password file <none>
I20260927 00:45:18.142011 10968 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:18.142072 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:18.142290 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:18.144333 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1ce954548094f0ea68000bc3c77c843"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-l5sb"
server_key: "74682a91f1f2255dcedd302fb5288cd7"
server_key_iv: "3b1d66cd13d2d12ed3138632409bad0e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.144871 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ce954548094f0ea68000bc3c77c843"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-l5sb"
server_key: "74682a91f1f2255dcedd302fb5288cd7"
server_key_iv: "3b1d66cd13d2d12ed3138632409bad0e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.148569 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:18.150954 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.152114 10968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:18.152235 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "c1ce954548094f0ea68000bc3c77c843"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-l5sb"
server_key: "74682a91f1f2255dcedd302fb5288cd7"
server_key_iv: "3b1d66cd13d2d12ed3138632409bad0e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.152338 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:18.179265 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:18.182152 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:18.182343 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:18.182904 10968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:18.183745 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:18.183794 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.183856 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:18.183894 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.192646 10968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37051
I20260927 00:45:18.192668 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37051 every 8 connection(s)
I20260927 00:45:18.193504 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:18.201686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 10968
I20260927 00:45:18.201838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:18.202167 392 external_mini_cluster.cc:1428] Setting key 5e4200bbdbd80f77e4f71a059f02a6fd
Sep 27 00:45:18 dist-test-slave-l5sb krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:18.207070 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.195227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59607 (local address 127.0.98.62:37047)
0927 00:45:18.195369 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.195373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.196194 (+ 821us) server_negotiation.cc:407] Connection header received
0927 00:45:18.197392 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.197395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.197440 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.197513 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.198642 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.199207 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.200268 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.200472 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.200551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.203171 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.203174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.203177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:18.203203 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.204941 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.205433 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.205436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.205438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.205519 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.205872 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.205877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.205879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.206060 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.206168 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.206758 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.206906 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:45:18.208091 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.195604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.3:59607)
0927 00:45:18.196049 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.196091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.197209 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.197669 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.197675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.198064 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:18.198517 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.198530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.199309 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.199312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.200160 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.200168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.200593 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.200596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.200746 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.201289 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:18.201306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:18.203000 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:18.205044 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.205049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.205060 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.205328 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.205604 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.205606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.205607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.205736 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.206175 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:18.206184 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:18.206496 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.206702 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.207017 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260927 00:45:18.209232 11097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 00:45:18.209432 11097 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:18.209913 11097 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
I20260927 00:45:18.210835 10638 ts_manager.cc:194] Registered new tserver with Master: c1ce954548094f0ea68000bc3c77c843 (127.0.98.3:37051)
I20260927 00:45:18.211403 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59607
I20260927 00:45:18.216017 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:18.225124 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.217935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.98.62:37047)
0927 00:45:18.218067 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.218072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.218198 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:45:18.218352 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.218355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.218399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.218478 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.219204 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.219712 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.220546 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.220766 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.220866 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.222345 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.222349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.222352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:18.222375 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.223526 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.223929 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.223934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.223936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.224014 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.224278 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.224281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.224283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.224487 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.224620 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.224851 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.224961 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:45:18.227749 10638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:18.230692 10638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:18.244745 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.239137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45159 (local address 127.0.0.1:60820)
0927 00:45:18.239367 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.239378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.239445 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.240492 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.240497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.240521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.240675 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.240681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.241755 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.241758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.243128 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.243136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.244202 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.244206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.244333 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.244375 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.244480 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.244593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260927 00:45:18.244755 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.239589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.98.2:45159)
0927 00:45:18.239949 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.239961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.239986 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:45:18.240065 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.240070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.240320 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.240512 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.240784 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.241625 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.243253 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.244066 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.244181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.244325 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.244520 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.244613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":193,"threads_started":1}
I20260927 00:45:18.245162 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.238900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.0.98.3:37051)
0927 00:45:18.240924 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.240931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.240947 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:45:18.241001 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.241004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.241132 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.241273 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.241513 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.242366 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.243631 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.244190 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.244569 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.244698 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.244744 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.245010 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":146,"threads_started":1}
I20260927 00:45:18.245329 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.238754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.0.1:46986)
0927 00:45:18.239237 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.239289 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.239419 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.241257 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.241260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.241278 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.241393 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.241401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.242491 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.242495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.243510 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.243519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.244316 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.244319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.244388 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.244404 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.244457 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.245164 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":597,"thread_start_us":86,"threads_started":1}
I20260927 00:45:18.245644 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.239528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33857 (local address 127.0.0.1:44128)
0927 00:45:18.239805 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.239818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.239906 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.241419 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.241424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.241444 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.241535 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.241540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.242961 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.242964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.244076 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.244085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.245199 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.245206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.245288 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.245395 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.245470 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.245531 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260927 00:45:18.245847 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.239793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.98.1:33857)
0927 00:45:18.240180 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.240195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.240218 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:45:18.240293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.240297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.240482 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.240632 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.241649 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.242814 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.244201 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.244922 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.245013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.245139 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.245506 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.245600 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":66,"threads_started":1}
I20260927 00:45:18.248088 10763 tablet_service.cc:1467] Processing CreateTablet for tablet 3556354185a14c789d0b0ee55edbec28 (DEFAULT_TABLE table=test-table [id=40764ffcf8534202ad4a958245be3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.248062 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 3556354185a14c789d0b0ee55edbec28 (DEFAULT_TABLE table=test-table [id=40764ffcf8534202ad4a958245be3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.248339 11031 tablet_service.cc:1467] Processing CreateTablet for tablet 3556354185a14c789d0b0ee55edbec28 (DEFAULT_TABLE table=test-table [id=40764ffcf8534202ad4a958245be3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.249038 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3556354185a14c789d0b0ee55edbec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.249073 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3556354185a14c789d0b0ee55edbec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.249131 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3556354185a14c789d0b0ee55edbec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.255506 11114 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: Bootstrap starting.
I20260927 00:45:18.256875 11116 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Bootstrap starting.
I20260927 00:45:18.257865 11114 tablet_bootstrap.cc:654] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.258095 11115 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Bootstrap starting.
I20260927 00:45:18.258936 11114 log.cc:824] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.259197 11116 tablet_bootstrap.cc:654] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.259979 11116 log.cc:824] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.260187 11115 tablet_bootstrap.cc:654] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.260900 11114 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: No bootstrap required, opened a new log
I20260927 00:45:18.261099 11114 ts_tablet_manager.cc:1397] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:45:18.261334 11115 log.cc:824] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.261929 11116 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: No bootstrap required, opened a new log
I20260927 00:45:18.262115 11116 ts_tablet_manager.cc:1397] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:45:18.263185 11115 tablet_bootstrap.cc:492] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: No bootstrap required, opened a new log
I20260927 00:45:18.263371 11115 ts_tablet_manager.cc:1397] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:18.264211 11114 raft_consensus.cc:348] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.264442 11114 raft_consensus.cc:374] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.264498 11114 raft_consensus.cc:729] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8b7e208bcc4169940149f725f65e41, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.264926 11114 consensus_queue.cc:260] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.265542 10829 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 00:45:18.265913 11114 ts_tablet_manager.cc:1428] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:45:18.265857 11116 raft_consensus.cc:348] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.266093 11116 raft_consensus.cc:374] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.266137 11116 raft_consensus.cc:729] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c0c438538a4cb78f03010b39738933, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.266556 11116 consensus_queue.cc:260] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.267006 11115 raft_consensus.cc:348] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.267242 11115 raft_consensus.cc:374] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.267287 11115 raft_consensus.cc:729] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ce954548094f0ea68000bc3c77c843, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.267478 11116 ts_tablet_manager.cc:1428] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:45:18.267690 11115 consensus_queue.cc:260] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.267774 10963 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 00:45:18.268388 11097 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 00:45:18.268579 11115 ts_tablet_manager.cc:1428] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:45:18.313081 11122 raft_consensus.cc:482] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:18.313299 11122 raft_consensus.cc:504] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.314575 11122 leader_election.cc:288] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad8b7e208bcc4169940149f725f65e41 (127.0.98.1:33857), 48c0c438538a4cb78f03010b39738933 (127.0.98.2:45159)
I20260927 00:45:18.318356 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.314942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33627 (local address 127.0.98.1:33857)
0927 00:45:18.315063 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.315070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.315088 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:45:18.315226 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.315229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.315288 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.315360 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.315708 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.316224 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.317327 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.317749 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.317930 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.317992 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.318172 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.318240 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:45:18.318357 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.314765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33857 (local address 127.0.98.3:33627)
0927 00:45:18.314988 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.315004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.315087 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.315484 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.315486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.315503 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.315599 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.315604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.316380 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.316383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.317205 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.317214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.317938 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.317941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.318047 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.318063 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.318146 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.318222 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":38}
I20260927 00:45:18.318605 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.315247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35567 (local address 127.0.98.2:45159)
0927 00:45:18.315369 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.315374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.315387 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:18.315473 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.315477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.315525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.315608 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.315859 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.316464 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.317588 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.318058 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.318169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.318229 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.318411 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.318493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:45:18.318954 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.314833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45159 (local address 127.0.98.3:35567)
0927 00:45:18.315251 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.315265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.315380 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.315615 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.315618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.315633 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.315749 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.315755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.316564 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.316570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.317467 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.317482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.318171 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.318174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.318286 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.318312 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.318395 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.318841 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
I20260927 00:45:18.318976 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3556354185a14c789d0b0ee55edbec28" candidate_uuid: "c1ce954548094f0ea68000bc3c77c843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8b7e208bcc4169940149f725f65e41" is_pre_election: true
I20260927 00:45:18.319315 10783 raft_consensus.cc:2454] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ce954548094f0ea68000bc3c77c843 in term 0.
I20260927 00:45:18.319595 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3556354185a14c789d0b0ee55edbec28" candidate_uuid: "c1ce954548094f0ea68000bc3c77c843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c0c438538a4cb78f03010b39738933" is_pre_election: true
I20260927 00:45:18.319790 10985 leader_election.cc:302] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8b7e208bcc4169940149f725f65e41, c1ce954548094f0ea68000bc3c77c843; no voters:
I20260927 00:45:18.319901 10917 raft_consensus.cc:2454] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1ce954548094f0ea68000bc3c77c843 in term 0.
I20260927 00:45:18.320060 11122 raft_consensus.cc:2781] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:18.320158 11122 raft_consensus.cc:482] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:18.320217 11122 raft_consensus.cc:3037] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.321239 11122 raft_consensus.cc:504] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.321576 11122 leader_election.cc:288] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [CANDIDATE]: Term 1 election: Requested vote from peers ad8b7e208bcc4169940149f725f65e41 (127.0.98.1:33857), 48c0c438538a4cb78f03010b39738933 (127.0.98.2:45159)
I20260927 00:45:18.321893 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3556354185a14c789d0b0ee55edbec28" candidate_uuid: "c1ce954548094f0ea68000bc3c77c843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c0c438538a4cb78f03010b39738933"
I20260927 00:45:18.322022 10917 raft_consensus.cc:3037] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.322029 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3556354185a14c789d0b0ee55edbec28" candidate_uuid: "c1ce954548094f0ea68000bc3c77c843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8b7e208bcc4169940149f725f65e41"
I20260927 00:45:18.322158 10783 raft_consensus.cc:3037] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.323071 10917 raft_consensus.cc:2454] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ce954548094f0ea68000bc3c77c843 in term 1.
I20260927 00:45:18.323154 10783 raft_consensus.cc:2454] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1ce954548094f0ea68000bc3c77c843 in term 1.
I20260927 00:45:18.323350 10986 leader_election.cc:302] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c0c438538a4cb78f03010b39738933, c1ce954548094f0ea68000bc3c77c843; no voters:
I20260927 00:45:18.323542 11122 raft_consensus.cc:2781] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:18.323735 11122 raft_consensus.cc:686] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 1 LEADER]: Becoming Leader. State: Replica: c1ce954548094f0ea68000bc3c77c843, State: Running, Role: LEADER
I20260927 00:45:18.324079 11122 consensus_queue.cc:237] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } }
I20260927 00:45:18.327147 10638 catalog_manager.cc:5495] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1ce954548094f0ea68000bc3c77c843 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c1ce954548094f0ea68000bc3c77c843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ce954548094f0ea68000bc3c77c843" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 } health_report { overall_health: UNKNOWN } } }
I20260927 00:45:18.342000 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.98.3:37051)
0927 00:45:18.338696 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.338700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.338713 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:18.338852 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.338855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.338908 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.338992 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:18.339324 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.339867 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.340698 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.340911 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.340988 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.341222 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:18.341617 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:18.341705 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.341756 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.341852 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:45:18.347590 10783 raft_consensus.cc:1261] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 1 FOLLOWER]: Refusing update from remote peer c1ce954548094f0ea68000bc3c77c843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:18.347590 10917 raft_consensus.cc:1261] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 1 FOLLOWER]: Refusing update from remote peer c1ce954548094f0ea68000bc3c77c843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:18.348279 11122 consensus_queue.cc:1035] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8b7e208bcc4169940149f725f65e41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:18.348502 11124 consensus_queue.cc:1035] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c0c438538a4cb78f03010b39738933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:18.355000 11127 mvcc.cc:205] Tried to move back new op lower bound from 7333764785545080832 to 7333764785458163712. Current Snapshot: MvccSnapshot[applied={T|T < 7333764785545080832}]
I20260927 00:45:18.358126 11129 mvcc.cc:205] Tried to move back new op lower bound from 7333764785545080832 to 7333764785458163712. Current Snapshot: MvccSnapshot[applied={T|T < 7333764785545080832}]
I20260927 00:45:18.364311 10636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:18.364455 10636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:18.366691 10636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10048b011b93435f9ab916da11420f21: Sending DeleteTablet for 3 replicas of tablet 3556354185a14c789d0b0ee55edbec28
I20260927 00:45:18.367514 10763 tablet_service.cc:1514] Processing DeleteTablet for tablet 3556354185a14c789d0b0ee55edbec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44128
I20260927 00:45:18.367549 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3556354185a14c789d0b0ee55edbec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60820
I20260927 00:45:18.367655 11031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3556354185a14c789d0b0ee55edbec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46986
I20260927 00:45:18.367995 11138 tablet_replica.cc:330] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41: stopping tablet replica
I20260927 00:45:18.368317 11137 tablet_replica.cc:330] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: stopping tablet replica
I20260927 00:45:18.368342 11138 raft_consensus.cc:2229] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:18.368599 11137 raft_consensus.cc:2229] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:18.368644 11138 raft_consensus.cc:2258] T 3556354185a14c789d0b0ee55edbec28 P ad8b7e208bcc4169940149f725f65e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.368731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10700
I20260927 00:45:18.368884 11137 raft_consensus.cc:2258] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.369316 11139 tablet_replica.cc:330] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: stopping tablet replica
I20260927 00:45:18.369568 11139 raft_consensus.cc:2229] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:18.369977 11139 raft_consensus.cc:2258] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.370522 11137 ts_tablet_manager.cc:1905] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:18.372133 11139 ts_tablet_manager.cc:1905] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:18.373427 11137 ts_tablet_manager.cc:1918] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:18.373536 11137 log.cc:1196] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/3556354185a14c789d0b0ee55edbec28
I20260927 00:45:18.373886 11137 ts_tablet_manager.cc:1939] T 3556354185a14c789d0b0ee55edbec28 P 48c0c438538a4cb78f03010b39738933: Deleting consensus metadata
I20260927 00:45:18.375087 11139 ts_tablet_manager.cc:1918] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:18.375178 11139 log.cc:1196] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/3556354185a14c789d0b0ee55edbec28
I20260927 00:45:18.375176 10624 catalog_manager.cc:4841] TS 48c0c438538a4cb78f03010b39738933 (127.0.98.2:45159): tablet 3556354185a14c789d0b0ee55edbec28 (table test-table [id=40764ffcf8534202ad4a958245be3355]) successfully deleted
I20260927 00:45:18.375514 11139 ts_tablet_manager.cc:1939] T 3556354185a14c789d0b0ee55edbec28 P c1ce954548094f0ea68000bc3c77c843: Deleting consensus metadata
I20260927 00:45:18.376598 10623 catalog_manager.cc:4841] TS c1ce954548094f0ea68000bc3c77c843 (127.0.98.3:37051): tablet 3556354185a14c789d0b0ee55edbec28 (table test-table [id=40764ffcf8534202ad4a958245be3355]) successfully deleted
W20260927 00:45:18.378885 10623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33857 (error 108)
I20260927 00:45:18.379004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10834
I20260927 00:45:18.380512 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.380206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33857 (local address 127.0.0.1:44134)
0927 00:45:18.380372 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.380431 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 00:45:18.380661 10623 catalog_manager.cc:4584] TS ad8b7e208bcc4169940149f725f65e41 (127.0.98.1:33857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3556354185a14c789d0b0ee55edbec28: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33857: connect: Connection refused (error 111)
I20260927 00:45:18.384920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10968
I20260927 00:45:18.391043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 10606
2026-09-27T00:45:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2322 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:18 dist-test-slave-l5sb krb5kdc[11143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:18 dist-test-slave-l5sb krb5kdc[11143](info): set up 2 sockets
Sep 27 00:45:18 dist-test-slave-l5sb krb5kdc[11143](info): commencing operation
krb5kdc: starting...
W20260927 00:45:19.438943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:19 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:45:19.608119 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjP59Ka/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:45:22.148 UTC [11173] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:45:22.149 UTC [11173] LOG: listening on IPv4 address "127.0.98.20", port 58717
2026-09-27 00:45:22.152 UTC [11173] LOG: listening on Unix socket "/tmp/.s.PGSQL.58717"
2026-09-27 00:45:22.187 UTC [11176] LOG: database system was shut down at 2026-09-27 00:45:20 UTC
2026-09-27 00:45:22.191 UTC [11173] LOG: database system is ready to accept connections
I20260927 00:45:23.143204 392 mini_postgres.cc:96] Postgres bound to 58717
2026-09-27 00:45:23.149 UTC [11184] FATAL: database "slave" does not exist
127.0.98.20:58717 - accepting connections
I20260927 00:45:23.150185 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:45:23.165201 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:45:23.665216 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:45:23.665315 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin
2026-09-27 00:45:23,945 [I] DB FLAVOR :POSTGRES
2026-09-27 00:45:23,945 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:45:23,945 [I] Checking connection..
2026-09-27 00:45:23,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:45:24,197 [I] Checking connection passed.
2026-09-27 00:45:24,197 [I] --------- Verifying version history table ---------
2026-09-27 00:45:24,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:45:24,452 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:45:24,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:45:24,689 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:45:24,690 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:45:24,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:45:24.927 UTC [11248] WARNING: there is no transaction in progress
2026-09-27 00:45:24,933 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:45:24,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:45:25,175 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:45:25,176 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:45:25,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:45:25,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:45:25,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l5sb.c.gcp-upstream.internal','N') ;"
2026-09-27 00:45:25,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:45:26,139 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:45:26,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:45:26,402 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:45:26,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:45:26,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:45:26,869 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:45:26,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:45:27.827 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.843 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.861 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.876 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.893 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.913 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.973 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.981 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:27.996 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.006 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.016 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.030 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.039 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.045 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.051 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28.058 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 00:45:28,706 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:45:28,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:45:28,948 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:45:28,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:45:29,210 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:45:29,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:45:29,461 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:45:29,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:45:29,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:45:29,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l5sb.c.gcp-upstream.internal';"
2026-09-27 00:45:30,171 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:45:30,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:45:30,402 [I] DB_PATCHES have already been applied
I20260927 00:45:30.407960 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjP59Ka/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:45:30.408039 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:45:30.411412 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40917
Sep 27, 2026 12:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:45:31.574062 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.424s user 0.003s sys 0.007s
Sep 27, 2026 12:45:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:45:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 12:45:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:45:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:45:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:45:51.789850 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:45:51 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:45:51.876780 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:45:51.885938 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:45:52.757861 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:45:52.757941 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms
I20260927 00:45:52.758036 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58717
2026-09-27 00:45:52,944 [I] DB FLAVOR :POSTGRES
2026-09-27 00:45:52,945 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:45:52,945 [I] Checking connection
2026-09-27 00:45:52,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:45:53,179 [I] connection success
2026-09-27 00:45:53,179 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:45:53,179 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:45:53,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:45:53,439 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:45:53,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:45:53,439 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:45:53,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:45:53,715 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:45:54.194911 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:45:54.367478 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:45:54.412448 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:45:54 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:45:54.808696 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:45:54.808853 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:45:54.808874 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:45:54.812882 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11536
I20260927 00:45:54.813043 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34571
Sep 27, 2026 12:45:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:45:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:45:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:45:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:45:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:45:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 00:45:56 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:57.098529 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.222s user 0.019s sys 0.152s
I20260927 00:45:57.098673 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:45:57.098781 392 mini_ranger_kms.cc:401] 127.0.98.20:34571/kms/v1/keys
Sep 27 00:45:57 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:45:57 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:45:59 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:59.899063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37837
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34571/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:00.031554 11587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:00.031805 11587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:00.031843 11587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:00.035704 11587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:00.035755 11587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:00.035774 11587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:00.035790 11587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:00.035805 11587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:00.041088 11587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34571/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37837
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:00.042074 11587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:00.043022 11587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:46:00.051600 11587 server_base.cc:1034] running on GCE node
W20260927 00:46:00.051674 11593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:00.051716 11592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:00.051613 11595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:00.052421 11587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:00.053457 11587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:00.054610 11587 hybrid_clock.cc:648] HybridClock initialized: now 1790469960054572 us; error 52 us; skew 500 ppm
Sep 27 00:46:00 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:00.069576 11587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:00.070711 11587 webserver.cc:460] Webserver started at http://127.0.98.62:37749/ using document root <none> and password file <none>
I20260927 00:46:00.071306 11587 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:00.071365 11587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:00.071596 11587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:00 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:00.129441 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "534ce93783b04eb092f0277a6e5e96a1"
format_stamp: "Formatted at 2026-09-27 00:46:00 on dist-test-slave-l5sb"
server_key: "22a53c79e79257175e47cf3992b43d89"
server_key_iv: "58a12915875b559cda4d52dc798309ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:00.130235 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "534ce93783b04eb092f0277a6e5e96a1"
format_stamp: "Formatted at 2026-09-27 00:46:00 on dist-test-slave-l5sb"
server_key: "22a53c79e79257175e47cf3992b43d89"
server_key_iv: "58a12915875b559cda4d52dc798309ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:00.134991 11587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:46:00.152709 11602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:00.154407 11587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:00.154549 11587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "534ce93783b04eb092f0277a6e5e96a1"
format_stamp: "Formatted at 2026-09-27 00:46:00 on dist-test-slave-l5sb"
server_key: "22a53c79e79257175e47cf3992b43d89"
server_key_iv: "58a12915875b559cda4d52dc798309ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:00.154690 11587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:00.190575 11587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:00.193375 11587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:00.193533 11587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:00.200990 11587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37837
I20260927 00:46:00.201014 11654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37837 every 8 connection(s)
I20260927 00:46:00.201999 11587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:00.204641 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:00.208811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 11587
I20260927 00:46:00.208910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:00.209193 392 external_mini_cluster.cc:1428] Setting key 088f1653cdb87d3d746de513b89e17a3
I20260927 00:46:00.211011 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: Bootstrap starting.
I20260927 00:46:00.213140 11655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:00.213948 11655 log.cc:824] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:00.215683 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: No bootstrap required, opened a new log
Sep 27 00:46:00 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:00.219835 11655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260927 00:46:00.220098 11655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:00.220192 11655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534ce93783b04eb092f0277a6e5e96a1, State: Initialized, Role: FOLLOWER
I20260927 00:46:00.220656 11655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260927 00:46:00.220773 11655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:00.220853 11655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:00.220947 11655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:00.222023 11655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260927 00:46:00.222378 11655 leader_election.cc:302] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534ce93783b04eb092f0277a6e5e96a1; no voters:
I20260927 00:46:00.222723 11655 leader_election.cc:288] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:00.222820 11660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:00.223097 11660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [term 1 LEADER]: Becoming Leader. State: Replica: 534ce93783b04eb092f0277a6e5e96a1, State: Running, Role: LEADER
I20260927 00:46:00.223438 11660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260927 00:46:00.224009 11655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:00.224048 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:00.210693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.0.98.62:37837)
0927 00:46:00.211067 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:00.211084 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:00.211157 (+ 73us) server_negotiation.cc:407] Connection header received
0927 00:46:00.211782 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:00.211799 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:00.212110 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:00.212451 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:00.213213 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:00.214698 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:00.215537 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:00.216052 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:00.216218 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:00.218963 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:00.218968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:00.218980 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:00.219004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:00.220723 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:00.221358 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:00.221363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:00.221367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:00.221475 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:00.221825 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:00.221828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:00.221830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:00.222292 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:00.222437 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:00.222778 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 00:46:00.223081 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260927 00:46:00.224689 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534ce93783b04eb092f0277a6e5e96a1. Latest consensus state: current_term: 1 leader_uuid: "534ce93783b04eb092f0277a6e5e96a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } } }
I20260927 00:46:00.224797 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:00.225139 11667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:00.226747 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534ce93783b04eb092f0277a6e5e96a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ce93783b04eb092f0277a6e5e96a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } } }
I20260927 00:46:00.226917 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:00.228381 11667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:00.228818 11655 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:46:00.238601 11667 catalog_manager.cc:1347] Generated new cluster ID: 4f964c09eeff4637b3dce56a410cf388
I20260927 00:46:00.238695 11667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:00.247431 11667 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:00.248777 11667 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:00.259459 11667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: Generated new TSK 0
I20260927 00:46:00.260715 11667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:00.929350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:01.073006 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:01.073443 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:01.073523 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:01.079705 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:01.079834 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:01.079957 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:01.087102 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:01.088228 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:01.089478 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:01.101410 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:01.101661 11700 server_base.cc:1034] running on GCE node
W20260927 00:46:01.101413 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:01.101969 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:01.102428 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:01.103125 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:01.104315 11700 hybrid_clock.cc:648] HybridClock initialized: now 1790469961104284 us; error 46 us; skew 500 ppm
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:01.107164 11700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:01.108119 11700 webserver.cc:460] Webserver started at http://127.0.98.1:35411/ using document root <none> and password file <none>
I20260927 00:46:01.108666 11700 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:01.108754 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:01.108963 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:01.122890 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "dab611250ed843e7b38174881464623e"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "e003287fa2154790d5d98e13c3a8805b"
server_key_iv: "a7c121a1a8a4f99fa7d2a98669f156e8"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.123783 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dab611250ed843e7b38174881464623e"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "e003287fa2154790d5d98e13c3a8805b"
server_key_iv: "a7c121a1a8a4f99fa7d2a98669f156e8"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.129034 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 00:46:01.150650 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.152410 11700 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 00:46:01.152542 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "dab611250ed843e7b38174881464623e"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "e003287fa2154790d5d98e13c3a8805b"
server_key_iv: "a7c121a1a8a4f99fa7d2a98669f156e8"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.152678 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:01.176733 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:01.182175 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:01.182433 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:01.183187 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:01.184293 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:01.184397 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.184521 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:01.184602 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.196185 11700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41935
I20260927 00:46:01.196270 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41935 every 8 connection(s)
I20260927 00:46:01.197366 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:01.210417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 11700
I20260927 00:46:01.210549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:01.210857 392 external_mini_cluster.cc:1428] Setting key ca290255883f6dbafff3a439e982aa71
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:01.234266 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.200313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.1:41291)
0927 00:46:01.202086 (+ 1773us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:01.202144 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:01.203197 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:01.220638 (+ 17441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:01.220650 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:01.221085 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:01.221870 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:01.221892 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.223045 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.223049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:01.224446 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:01.224462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.225209 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.225213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:01.225902 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:01.226604 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:01.226625 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:01.228217 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:01.230845 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:01.230851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:01.230869 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:01.231240 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:01.231574 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:01.231577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:01.231579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:01.231776 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:01.232283 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:01.232290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:01.232634 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 00:46:01.232837 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:46:01.233185 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"thread_start_us":140,"threads_started":1}
I20260927 00:46:01.235162 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.209932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41291 (local address 127.0.98.62:37837)
0927 00:46:01.220066 (+ 10134us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:01.220077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:01.220104 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:46:01.220156 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:01.220159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:01.220217 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:01.222026 (+ 1809us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:01.222137 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.222893 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.224625 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.224945 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.225036 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:01.228412 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:01.228416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:01.228419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:01.228444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:01.230704 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:01.231365 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:01.231368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:01.231370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:01.231463 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:01.231891 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:01.231894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:01.231896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:01.232133 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:01.232228 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:01.234807 (+ 2579us) server_negotiation.cc:299] Negotiation successful
0927 00:46:01.234976 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10040,"thread_start_us":126,"threads_started":1}
I20260927 00:46:01.236295 11833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260927 00:46:01.236563 11833 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:01.237162 11833 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260927 00:46:01.238942 11619 ts_manager.cc:194] Registered new tserver with Master: dab611250ed843e7b38174881464623e (127.0.98.1:41935)
I20260927 00:46:01.240427 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:01.258241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:46:01.449568 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:01.450040 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:01.450150 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:01.455792 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:01.455871 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:01.455955 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:01.463485 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:01.465077 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:01.466439 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:01.478916 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:01.479570 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:01.479846 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:01.480836 11839 server_base.cc:1034] running on GCE node
I20260927 00:46:01.481448 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:01.482249 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:01.483409 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790469961483340 us; error 83 us; skew 500 ppm
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:01.487107 11839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:01.488273 11839 webserver.cc:460] Webserver started at http://127.0.98.2:32835/ using document root <none> and password file <none>
I20260927 00:46:01.489017 11839 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:01.489151 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:01.489432 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:01.505903 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "8aa244080d264bf9bd62a992de2658c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "95502ad462800dec35edeaed051df7dd"
server_key_iv: "4f17aa7b4cec8cefcc68c5c1ce43b577"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.506758 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8aa244080d264bf9bd62a992de2658c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "95502ad462800dec35edeaed051df7dd"
server_key_iv: "4f17aa7b4cec8cefcc68c5c1ce43b577"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.515240 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.004s
I20260927 00:46:01.531291 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.532970 11839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:01.533155 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "8aa244080d264bf9bd62a992de2658c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "95502ad462800dec35edeaed051df7dd"
server_key_iv: "4f17aa7b4cec8cefcc68c5c1ce43b577"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.533371 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:01.550735 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:01.554225 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:01.554455 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:01.555096 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:01.556105 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:01.556222 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.556314 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:01.556375 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.570281 11839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45385
I20260927 00:46:01.571517 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:01.573012 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45385 every 8 connection(s)
I20260927 00:46:01.576936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 11839
I20260927 00:46:01.577064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:01.577394 392 external_mini_cluster.cc:1428] Setting key bf7a00fe48aa27c61fc7c0c72f37ddf7
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:46:01.602860 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.576335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42223 (local address 127.0.98.62:37837)
0927 00:46:01.576483 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:01.576488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:01.581597 (+ 5109us) server_negotiation.cc:407] Connection header received
0927 00:46:01.582881 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:01.582885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:01.582936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:01.582989 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:01.584649 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.585486 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.587228 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.587611 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.587729 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:01.598293 (+ 10564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:01.598298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:01.598301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:01.598334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:01.600190 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:01.600910 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:01.600914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:01.600916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:01.601003 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:01.601432 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:01.601435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:01.601437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:01.601664 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:01.601748 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:01.602527 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 00:46:01.602665 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:46:01.606480 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.575241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.2:42223)
0927 00:46:01.576659 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:01.581481 (+ 4822us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:01.582632 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:01.583427 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:01.583437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:01.583797 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:01.584480 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:01.584499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.585646 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.585649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:01.587080 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:01.587091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:01.587898 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:01.587902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:01.588101 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:01.594589 (+ 6488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:01.594612 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:01.598048 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:01.600407 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:01.600413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:01.600432 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:01.600788 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:01.601130 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:01.601133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:01.601135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:01.601321 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:01.601837 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:01.601844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:01.602207 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 00:46:01.604909 (+ 2702us) client_negotiation.cc:240] Negotiation successful
0927 00:46:01.605297 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":138,"threads_started":1}
I20260927 00:46:01.608109 11969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260927 00:46:01.608357 11969 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:01.608939 11969 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260927 00:46:01.610173 11619 ts_manager.cc:194] Registered new tserver with Master: 8aa244080d264bf9bd62a992de2658c0 (127.0.98.2:45385)
I20260927 00:46:01.610925 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:01.637885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:01.809851 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:01.810319 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:01.810449 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:01.820729 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:01.820883 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:01.821030 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:01.829435 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:01.831110 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:01.832451 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:01.845232 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:01.845248 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:01.846834 11974 server_base.cc:1034] running on GCE node
W20260927 00:46:01.847865 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:01.848398 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:01.849191 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:01.864689 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790469961864598 us; error 116 us; skew 500 ppm
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:01.870533 11974 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:01.871800 11974 webserver.cc:460] Webserver started at http://127.0.98.3:36181/ using document root <none> and password file <none>
I20260927 00:46:01.872587 11974 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:01.872761 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:01.873055 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:01 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:01.896205 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "8803fa78f46540ac96e07fdb9951f6c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "0a236a108cc433d9a1d6ad2443d41a98"
server_key_iv: "bbac3ef54860e79c9b1523376b944fe9"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.897468 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8803fa78f46540ac96e07fdb9951f6c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "0a236a108cc433d9a1d6ad2443d41a98"
server_key_iv: "bbac3ef54860e79c9b1523376b944fe9"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.902596 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 00:46:01.922435 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.924198 11974 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 00:46:01.924335 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "8803fa78f46540ac96e07fdb9951f6c0"
format_stamp: "Formatted at 2026-09-27 00:46:01 on dist-test-slave-l5sb"
server_key: "0a236a108cc433d9a1d6ad2443d41a98"
server_key_iv: "bbac3ef54860e79c9b1523376b944fe9"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:01.924464 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:01.961046 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:01.964689 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:01.964931 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:01.965626 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:01.966682 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:01.966784 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.966877 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:01.966941 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:01.977350 11974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41087
I20260927 00:46:01.991076 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:01.988430 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41087 every 8 connection(s)
I20260927 00:46:01.998063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 11974
I20260927 00:46:01.998201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:01.998492 392 external_mini_cluster.cc:1428] Setting key 2009403aa6ee19f38bfc870e69fe30b2
Sep 27 00:46:02 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:02.012969 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.996418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49865 (local address 127.0.98.62:37837)
0927 00:46:01.996539 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:01.996544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:01.996998 (+ 454us) server_negotiation.cc:407] Connection header received
0927 00:46:01.998255 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:01.998259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:01.998310 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:01.998360 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:02.001604 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.002469 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.004123 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.004466 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.004537 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.007500 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:02.007503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:02.007506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:02.007532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:02.009217 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:02.009986 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:02.009989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:02.009991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:02.010082 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:02.010551 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:02.010554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:02.010555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:02.010759 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:02.010834 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.012683 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.012813 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:46:02.014873 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:01.995301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.3:49865)
0927 00:46:01.996840 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:01.996895 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:01.998004 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:01.999385 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:01.999402 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.000699 (+ 1297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:02.001444 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.001463 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.002623 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.002626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.003981 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.003991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.004732 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.004735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.004920 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.005616 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:02.005636 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:02.007318 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:02.009353 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:02.009358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:02.009375 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:02.009708 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:02.010275 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:02.010278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:02.010280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:02.010445 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:02.011447 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:02.011453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:02.012410 (+ 957us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.013311 (+ 901us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.013840 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":166,"threads_started":1}
I20260927 00:46:02.016149 12103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260927 00:46:02.016368 12103 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:02.016880 12103 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260927 00:46:02.017956 11619 ts_manager.cc:194] Registered new tserver with Master: 8803fa78f46540ac96e07fdb9951f6c0 (127.0.98.3:41087)
I20260927 00:46:02.018618 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49865
I20260927 00:46:02.023988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:02.035521 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.026645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.0.98.62:37837)
0927 00:46:02.026745 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.026750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.026763 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:46:02.026859 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.026862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.026902 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.026981 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:02.027847 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.028573 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.029659 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.030009 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.030092 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.031849 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:02.031852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:02.031855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:02.031879 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:02.033598 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:02.034140 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:02.034143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:02.034145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:02.034231 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:02.034539 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:02.034542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:02.034543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:02.034737 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:02.034863 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.035116 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.035229 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:46:02.038668 11619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:02.042809 11619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:02.073091 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.056123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41935 (local address 127.0.0.1:51478)
0927 00:46:02.059821 (+ 3698us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.059836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:02.059919 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:02.064539 (+ 4620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:02.064542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.064560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:02.064671 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.064677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.067117 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.067121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.068366 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.071239 (+ 2873us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.072503 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.072507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.072638 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.072678 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.072811 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.072930 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3641,"thread_start_us":92,"threads_started":1}
I20260927 00:46:02.075181 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.058392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.98.3:41087)
0927 00:46:02.059454 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.059467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.061028 (+ 1561us) server_negotiation.cc:407] Connection header received
0927 00:46:02.061113 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.061118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.061410 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.061567 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:02.062854 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.065053 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.070099 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.070944 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.071042 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.071200 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.074903 (+ 3703us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.075038 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":138,"threads_started":1}
I20260927 00:46:02.073873 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.057995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41087 (local address 127.0.0.1:42482)
0927 00:46:02.060836 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.060849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:02.060933 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:02.062546 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:02.062552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.062574 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:02.062724 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.062730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.065186 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.065189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.069930 (+ 4741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.069940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.073448 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.073452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.073542 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.073564 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.073631 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.073699 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2781,"thread_start_us":79,"threads_started":1}
I20260927 00:46:02.078262 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.053438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45385 (local address 127.0.0.1:56022)
0927 00:46:02.059148 (+ 5710us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.059203 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:02.059574 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:02.070674 (+ 11100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:02.070677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.070694 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:02.070777 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.070783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.072190 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.072193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.073267 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.073277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.077887 (+ 4610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.077891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.077986 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.078009 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.078060 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.078127 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5516,"thread_start_us":107,"threads_started":1}
I20260927 00:46:02.079129 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.055317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.0.98.2:45385)
0927 00:46:02.069998 (+ 14681us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.070016 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.070041 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:02.070120 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.070126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.070365 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.070524 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:02.070911 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.072049 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.073432 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.074313 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.078695 (+ 4382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.078848 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.078901 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.078992 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14550,"thread_start_us":101,"threads_started":1}
I20260927 00:46:02.079427 12037 tablet_service.cc:1467] Processing CreateTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 (DEFAULT_TABLE table=test-table [id=ffcb2d8df1484261b8e9d6a3b14455df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:02.086921 11901 tablet_service.cc:1467] Processing CreateTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 (DEFAULT_TABLE table=test-table [id=ffcb2d8df1484261b8e9d6a3b14455df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:02.088229 11901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d579762b914cd9a7292458a6cbb0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:02.090624 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d579762b914cd9a7292458a6cbb0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:02.107441 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.056539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.0.98.1:41935)
0927 00:46:02.061888 (+ 5349us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.061895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.061917 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:46:02.062065 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.062069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.062258 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.062388 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:02.065848 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.066970 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.071363 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.072202 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.072296 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.072444 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.107110 (+ 34666us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.107264 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5243,"thread_start_us":85,"threads_started":1}
I20260927 00:46:02.112479 12121 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Bootstrap starting.
I20260927 00:46:02.114341 11765 tablet_service.cc:1467] Processing CreateTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 (DEFAULT_TABLE table=test-table [id=ffcb2d8df1484261b8e9d6a3b14455df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:02.115490 11765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d579762b914cd9a7292458a6cbb0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:02.116047 12121 tablet_bootstrap.cc:654] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:02.116940 12121 log.cc:824] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:02.118775 12122 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Bootstrap starting.
I20260927 00:46:02.121055 12122 tablet_bootstrap.cc:654] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:02.121955 12122 log.cc:824] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:02.124274 12121 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: No bootstrap required, opened a new log
I20260927 00:46:02.124595 12121 ts_tablet_manager.cc:1397] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260927 00:46:02.129503 12121 raft_consensus.cc:348] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.130415 12121 raft_consensus.cc:374] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:02.130537 12121 raft_consensus.cc:729] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8803fa78f46540ac96e07fdb9951f6c0, State: Initialized, Role: FOLLOWER
I20260927 00:46:02.132757 12122 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: No bootstrap required, opened a new log
I20260927 00:46:02.134426 12122 ts_tablet_manager.cc:1397] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20260927 00:46:02.135885 12125 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Bootstrap starting.
I20260927 00:46:02.138422 12125 tablet_bootstrap.cc:654] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:02.139338 12125 log.cc:824] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:02.139691 12122 raft_consensus.cc:348] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.140407 12121 consensus_queue.cc:260] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.141847 12122 raft_consensus.cc:374] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:02.141978 12122 raft_consensus.cc:729] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa244080d264bf9bd62a992de2658c0, State: Initialized, Role: FOLLOWER
I20260927 00:46:02.142581 12122 consensus_queue.cc:260] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.143810 12122 ts_tablet_manager.cc:1428] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 00:46:02.144019 12121 ts_tablet_manager.cc:1428] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260927 00:46:02.144492 12103 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
I20260927 00:46:02.149257 11969 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
I20260927 00:46:02.155839 12125 tablet_bootstrap.cc:492] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: No bootstrap required, opened a new log
I20260927 00:46:02.156230 12125 ts_tablet_manager.cc:1397] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260927 00:46:02.161399 12125 raft_consensus.cc:348] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.161873 12125 raft_consensus.cc:374] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:02.161968 12125 raft_consensus.cc:729] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab611250ed843e7b38174881464623e, State: Initialized, Role: FOLLOWER
I20260927 00:46:02.162586 12125 consensus_queue.cc:260] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.163801 12125 ts_tablet_manager.cc:1428] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:46:02.164211 11833 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
W20260927 00:46:02.199118 11834 tablet.cc:2369] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:02.215770 12129 raft_consensus.cc:482] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:02.216058 12129 raft_consensus.cc:504] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.219213 12129 leader_election.cc:288] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8aa244080d264bf9bd62a992de2658c0 (127.0.98.2:45385), 8803fa78f46540ac96e07fdb9951f6c0 (127.0.98.3:41087)
I20260927 00:46:02.223650 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.218188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33753 (local address 127.0.98.2:45385)
0927 00:46:02.218322 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.218329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.219390 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 00:46:02.219593 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.219596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.219641 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.219683 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:02.220039 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.220769 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.222347 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.222955 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.223039 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.223108 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.223431 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.223520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:02.224270 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.217740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45385 (local address 127.0.98.1:33753)
0927 00:46:02.219295 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.219311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:02.219467 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:02.219816 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:02.219819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.219835 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:02.219928 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.219933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.220912 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.220915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.222204 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.222214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.223202 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.223205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.223291 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.223317 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.224034 (+ 717us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.224114 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"thread_start_us":126,"threads_started":1}
I20260927 00:46:02.224339 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.219057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52317 (local address 127.0.98.3:41087)
0927 00:46:02.219175 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.219180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.219656 (+ 476us) server_negotiation.cc:407] Connection header received
0927 00:46:02.219876 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.219879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.219928 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.219976 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:02.220367 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.221106 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.223089 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.223644 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.223737 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.223796 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.224138 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.224214 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:46:02.224884 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.218659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41087 (local address 127.0.98.1:52317)
0927 00:46:02.219559 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.219575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:02.219744 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:02.220113 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:02.220116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:02.220134 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:02.220235 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.220241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.221281 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.221284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.222948 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:02.222958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.223898 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.223902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:02.223998 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.224018 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:46:02.224664 (+ 646us) client_negotiation.cc:240] Negotiation successful
0927 00:46:02.224750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":108,"threads_started":1}
I20260927 00:46:02.225348 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d579762b914cd9a7292458a6cbb0f7" candidate_uuid: "dab611250ed843e7b38174881464623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aa244080d264bf9bd62a992de2658c0" is_pre_election: true
I20260927 00:46:02.225695 11909 raft_consensus.cc:2454] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab611250ed843e7b38174881464623e in term 0.
I20260927 00:46:02.225672 12057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d579762b914cd9a7292458a6cbb0f7" candidate_uuid: "dab611250ed843e7b38174881464623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8803fa78f46540ac96e07fdb9951f6c0" is_pre_election: true
I20260927 00:46:02.225996 12057 raft_consensus.cc:2454] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab611250ed843e7b38174881464623e in term 0.
I20260927 00:46:02.226374 11721 leader_election.cc:302] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aa244080d264bf9bd62a992de2658c0, dab611250ed843e7b38174881464623e; no voters:
I20260927 00:46:02.226934 12129 raft_consensus.cc:2781] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:02.227033 12129 raft_consensus.cc:482] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:02.227080 12129 raft_consensus.cc:3037] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:02.228384 12129 raft_consensus.cc:504] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.228721 12129 leader_election.cc:288] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [CANDIDATE]: Term 1 election: Requested vote from peers 8aa244080d264bf9bd62a992de2658c0 (127.0.98.2:45385), 8803fa78f46540ac96e07fdb9951f6c0 (127.0.98.3:41087)
I20260927 00:46:02.229380 12057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d579762b914cd9a7292458a6cbb0f7" candidate_uuid: "dab611250ed843e7b38174881464623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8803fa78f46540ac96e07fdb9951f6c0"
I20260927 00:46:02.229496 12057 raft_consensus.cc:3037] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:02.230846 12057 raft_consensus.cc:2454] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab611250ed843e7b38174881464623e in term 1.
I20260927 00:46:02.231238 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d579762b914cd9a7292458a6cbb0f7" candidate_uuid: "dab611250ed843e7b38174881464623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aa244080d264bf9bd62a992de2658c0"
I20260927 00:46:02.231360 11909 raft_consensus.cc:3037] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:02.232591 11909 raft_consensus.cc:2454] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab611250ed843e7b38174881464623e in term 1.
I20260927 00:46:02.234724 11721 leader_election.cc:302] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8803fa78f46540ac96e07fdb9951f6c0, dab611250ed843e7b38174881464623e; no voters:
I20260927 00:46:02.237915 12129 raft_consensus.cc:2781] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:02.238229 12129 raft_consensus.cc:686] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 1 LEADER]: Becoming Leader. State: Replica: dab611250ed843e7b38174881464623e, State: Running, Role: LEADER
I20260927 00:46:02.238651 12129 consensus_queue.cc:237] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } }
I20260927 00:46:02.242594 11618 catalog_manager.cc:5495] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e reported cstate change: term changed from 0 to 1, leader changed from <none> to dab611250ed843e7b38174881464623e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dab611250ed843e7b38174881464623e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab611250ed843e7b38174881464623e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 } health_report { overall_health: UNKNOWN } } }
W20260927 00:46:02.243858 12104 tablet.cc:2369] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:46:02 dist-test-slave-l5sb krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:02.262148 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.256368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.98.1:41935)
0927 00:46:02.256494 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:02.256499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:02.256568 (+ 69us) server_negotiation.cc:407] Connection header received
0927 00:46:02.256773 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:02.256777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:02.256847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:02.256952 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:02.257528 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.258364 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.259613 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:02.259910 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:02.260020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:02.260662 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:02.261259 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:02.261381 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:02.261840 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 00:46:02.261899 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:46:02.271706 11909 raft_consensus.cc:1261] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 1 FOLLOWER]: Refusing update from remote peer dab611250ed843e7b38174881464623e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:02.271706 12057 raft_consensus.cc:1261] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 1 FOLLOWER]: Refusing update from remote peer dab611250ed843e7b38174881464623e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:02.272625 12129 consensus_queue.cc:1035] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aa244080d264bf9bd62a992de2658c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:02.273438 12133 consensus_queue.cc:1035] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8803fa78f46540ac96e07fdb9951f6c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:02.284121 12136 mvcc.cc:205] Tried to move back new op lower bound from 7333764965454942208 to 7333764965333958656. Current Snapshot: MvccSnapshot[applied={T|T < 7333764965454942208}]
I20260927 00:46:02.287978 12137 mvcc.cc:205] Tried to move back new op lower bound from 7333764965454942208 to 7333764965333958656. Current Snapshot: MvccSnapshot[applied={T|T < 7333764965454942208}]
I20260927 00:46:02.295235 12138 mvcc.cc:205] Tried to move back new op lower bound from 7333764965454942208 to 7333764965333958656. Current Snapshot: MvccSnapshot[applied={T|T < 7333764965454942208}]
I20260927 00:46:02.306717 11618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:02.306945 11618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:02.310969 11618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 534ce93783b04eb092f0277a6e5e96a1: Sending DeleteTablet for 3 replicas of tablet b0d579762b914cd9a7292458a6cbb0f7
I20260927 00:46:02.313195 11765 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51478
I20260927 00:46:02.314529 12037 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42482
I20260927 00:46:02.314579 11901 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d579762b914cd9a7292458a6cbb0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56022
I20260927 00:46:02.315265 12146 tablet_replica.cc:330] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e: stopping tablet replica
I20260927 00:46:02.315546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 11700
I20260927 00:46:02.315614 12146 raft_consensus.cc:2229] T b0d579762b914cd9a7292458a6cbb0f7 P dab611250ed843e7b38174881464623e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:02.316313 12147 tablet_replica.cc:330] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: stopping tablet replica
I20260927 00:46:02.316802 12147 raft_consensus.cc:2229] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:02.317199 12147 raft_consensus.cc:2258] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:02.317662 12148 tablet_replica.cc:330] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: stopping tablet replica
I20260927 00:46:02.318157 12148 raft_consensus.cc:2229] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:02.318478 12148 raft_consensus.cc:2258] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:02.323716 12148 ts_tablet_manager.cc:1905] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:02.327571 12148 ts_tablet_manager.cc:1918] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:02.327775 12148 log.cc:1196] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/b0d579762b914cd9a7292458a6cbb0f7
I20260927 00:46:02.328204 12148 ts_tablet_manager.cc:1939] T b0d579762b914cd9a7292458a6cbb0f7 P 8aa244080d264bf9bd62a992de2658c0: Deleting consensus metadata
I20260927 00:46:02.330962 12147 ts_tablet_manager.cc:1905] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:02.331363 11606 catalog_manager.cc:4841] TS 8aa244080d264bf9bd62a992de2658c0 (127.0.98.2:45385): tablet b0d579762b914cd9a7292458a6cbb0f7 (table test-table [id=ffcb2d8df1484261b8e9d6a3b14455df]) successfully deleted
W20260927 00:46:02.332665 11603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41935 (error 108)
I20260927 00:46:02.333875 12147 ts_tablet_manager.cc:1918] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:02.337045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 11839
I20260927 00:46:02.337646 12147 log.cc:1196] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/b0d579762b914cd9a7292458a6cbb0f7
I20260927 00:46:02.338054 12147 ts_tablet_manager.cc:1939] T b0d579762b914cd9a7292458a6cbb0f7 P 8803fa78f46540ac96e07fdb9951f6c0: Deleting consensus metadata
I20260927 00:46:02.338901 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:02.338499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41935 (local address 127.0.0.1:51484)
0927 00:46:02.338715 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:02.338792 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 00:46:02.339095 11603 catalog_manager.cc:4584] TS dab611250ed843e7b38174881464623e (127.0.98.1:41935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0d579762b914cd9a7292458a6cbb0f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41935: connect: Connection refused (error 111)
I20260927 00:46:02.339403 11606 catalog_manager.cc:4841] TS 8803fa78f46540ac96e07fdb9951f6c0 (127.0.98.3:41087): tablet b0d579762b914cd9a7292458a6cbb0f7 (table test-table [id=ffcb2d8df1484261b8e9d6a3b14455df]) successfully deleted
I20260927 00:46:02.355222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 11974
I20260927 00:46:02.370270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 11587
I20260927 00:46:02.379587 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:46:02.724344 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:46:02.724458 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:46:03.069623 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:46:03.069 UTC [11173] LOG: received smart shutdown request
2026-09-27 00:46:03.073 UTC [11173] LOG: background worker "logical replication launcher" (PID 11182) exited with exit code 1
2026-09-27 00:46:03.074 UTC [11177] LOG: shutting down
2026-09-27 00:46:03.240 UTC [11173] LOG: database system is shut down
2026-09-27T00:46:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44960 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:03 dist-test-slave-l5sb krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:03 dist-test-slave-l5sb krb5kdc[12158](info): set up 2 sockets
Sep 27 00:46:03 dist-test-slave-l5sb krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260927 00:46:04.385828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:04 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:46:04.552338 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjP59Ka/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:46:07.126 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:46:07.126 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 40237
2026-09-27 00:46:07.130 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.40237"
2026-09-27 00:46:07.151 UTC [12191] LOG: database system was shut down at 2026-09-27 00:46:05 UTC
2026-09-27 00:46:07.155 UTC [12188] LOG: database system is ready to accept connections
I20260927 00:46:08.105752 392 mini_postgres.cc:96] Postgres bound to 40237
2026-09-27 00:46:08.112 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:40237 - accepting connections
I20260927 00:46:08.113422 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:46:08.127719 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:46:08.607319 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:46:08.607407 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin
2026-09-27 00:46:08,884 [I] DB FLAVOR :POSTGRES
2026-09-27 00:46:08,885 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:46:08,885 [I] Checking connection..
2026-09-27 00:46:08,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:46:09,113 [I] Checking connection passed.
2026-09-27 00:46:09,113 [I] --------- Verifying version history table ---------
2026-09-27 00:46:09,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:46:09,348 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:46:09,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:46:09,572 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:46:09,572 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:46:09,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:46:09.797 UTC [12263] WARNING: there is no transaction in progress
2026-09-27 00:46:09,803 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:46:09,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:46:10,046 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:46:10,046 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:46:10,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:46:10,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:46:10,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l5sb.c.gcp-upstream.internal','N') ;"
2026-09-27 00:46:10,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:46:11,035 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:46:11,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:46:11,300 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:46:11,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:46:11,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:46:11,771 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:46:11,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:46:12.733 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.749 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.764 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.780 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.795 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.813 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.866 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.872 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.886 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.896 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.906 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.920 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.930 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.937 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.944 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:12.950 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 00:46:13,562 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:46:13,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:46:13,799 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:46:13,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:46:14,054 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:46:14,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:46:14,310 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:46:14,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:46:14,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:46:14,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l5sb.c.gcp-upstream.internal';"
2026-09-27 00:46:15,035 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:46:15,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:46:15,274 [I] DB_PATCHES have already been applied
I20260927 00:46:15.280498 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjP59Ka/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:46:15.280567 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:46:15.283785 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:35337
Sep 27, 2026 12:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:46:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:46:16.383397 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.270s user 0.002s sys 0.007s
Sep 27, 2026 12:46:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:46:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:46:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:46:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:46:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:46:37.857977 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:46:37 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469997, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:46:37.952565 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:46:37.962100 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:46:38.845919 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:46:38.845988 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms
I20260927 00:46:38.846001 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40237
2026-09-27 00:46:39,023 [I] DB FLAVOR :POSTGRES
2026-09-27 00:46:39,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:46:39,024 [I] Checking connection
2026-09-27 00:46:39,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:46:39,281 [I] connection success
2026-09-27 00:46:39,281 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:46:39,281 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:46:39,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:46:39,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:46:39,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:46:39,507 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:46:39,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:46:39,752 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:46:40.195037 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:46:40.382932 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:46:40.426079 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:46:40 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469997, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:40.821130 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:46:40.821277 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjP59Ka/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjP59Ka/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:46:40.821300 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:46:40.825309 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12551
I20260927 00:46:40.825393 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33075
Sep 27, 2026 12:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:46:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:46:43.061614 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.109s user 0.000s sys 0.156s
I20260927 00:46:43.061731 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:46:43.061820 392 mini_ranger_kms.cc:401] 127.0.98.20:33075/kms/v1/keys
Sep 27 00:46:43 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:46:43 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:46:43 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470003, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:46:46 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:46.108641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43725
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:46.218102 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:46.218340 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:46.218377 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:46.222266 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:46.222314 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:46.222333 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:46.222349 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:46.222365 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:46.227612 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33075/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43725
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:46.228519 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:46.229422 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:46.238075 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:46.238067 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:46.238114 12602 server_base.cc:1034] running on GCE node
W20260927 00:46:46.238066 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:46.238766 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:46.239832 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:46.240985 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790470006240962 us; error 36 us; skew 500 ppm
Sep 27 00:46:46 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:46.243641 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:46.244485 12602 webserver.cc:460] Webserver started at http://127.0.98.62:37021/ using document root <none> and password file <none>
I20260927 00:46:46.244977 12602 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:46.245029 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:46.245183 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:46 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:46.292012 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "f7befba2315d43018d8694904b0316c2"
format_stamp: "Formatted at 2026-09-27 00:46:46 on dist-test-slave-l5sb"
server_key: "514b64f66b2d0a3e129923af861740b2"
server_key_iv: "0c040a63113be900355fc4528151e4cf"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:46.292743 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7befba2315d43018d8694904b0316c2"
format_stamp: "Formatted at 2026-09-27 00:46:46 on dist-test-slave-l5sb"
server_key: "514b64f66b2d0a3e129923af861740b2"
server_key_iv: "0c040a63113be900355fc4528151e4cf"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:46.297678 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:46:46.312659 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:46.314195 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:46:46.314327 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "f7befba2315d43018d8694904b0316c2"
format_stamp: "Formatted at 2026-09-27 00:46:46 on dist-test-slave-l5sb"
server_key: "514b64f66b2d0a3e129923af861740b2"
server_key_iv: "0c040a63113be900355fc4528151e4cf"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:46.314462 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:46.329958 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:46.332787 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:46.333002 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:46.340807 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43725
I20260927 00:46:46.340852 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43725 every 8 connection(s)
I20260927 00:46:46.341816 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:46.345121 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:46.345383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 12602
I20260927 00:46:46.345504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:46.345835 392 external_mini_cluster.cc:1428] Setting key 7b614edc4107201438b30985ac3d6a98
I20260927 00:46:46.352600 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Bootstrap starting.
Sep 27 00:46:46 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:46.355433 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:46.356415 12670 log.cc:824] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:46.359004 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: No bootstrap required, opened a new log
I20260927 00:46:46.360607 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:46.347165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.0.98.62:43725)
0927 00:46:46.347546 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:46.347567 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:46.347795 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:46:46.348572 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:46.348588 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:46.348911 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:46.349236 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:46.350098 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:46.351080 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:46.351947 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:46.352449 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:46.352614 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:46.355346 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:46.355352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:46.355364 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:46.355389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:46.357869 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:46.358293 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:46.358299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:46.358305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:46.358410 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:46.358675 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:46.358680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:46.358681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:46.359074 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:46.359264 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:46.359564 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 00:46:46.359847 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260927 00:46:46.363130 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:46.363432 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:46.363502 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7befba2315d43018d8694904b0316c2, State: Initialized, Role: FOLLOWER
I20260927 00:46:46.363987 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:46.364163 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:46.364260 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:46.364363 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:46.365559 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:46.365962 12670 leader_election.cc:302] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7befba2315d43018d8694904b0316c2; no voters:
I20260927 00:46:46.366317 12670 leader_election.cc:288] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:46.366760 12676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:46.367015 12676 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 LEADER]: Becoming Leader. State: Replica: f7befba2315d43018d8694904b0316c2, State: Running, Role: LEADER
I20260927 00:46:46.367319 12676 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:46.367659 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:46.368451 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7befba2315d43018d8694904b0316c2. Latest consensus state: current_term: 1 leader_uuid: "f7befba2315d43018d8694904b0316c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } } }
I20260927 00:46:46.368582 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:46.368990 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7befba2315d43018d8694904b0316c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } } }
I20260927 00:46:46.369104 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:46.370005 12681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:46.370275 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:46:46.376920 12681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:46.383883 12681 catalog_manager.cc:1347] Generated new cluster ID: c6e195b6ee37482f948ee284a3515e40
I20260927 00:46:46.383947 12681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:46.406945 12681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:46.408458 12681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:46.426504 12681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Generated new TSK 0
I20260927 00:46:46.427551 12681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:47.097762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:47.235236 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:47.235579 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:47.235687 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:47.240300 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:47.240398 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:47.240492 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:47.247321 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:47.248476 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:47.249539 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:47.260901 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.260901 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.261031 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.262032 12715 server_base.cc:1034] running on GCE node
I20260927 00:46:47.262468 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:47.263161 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:47.264351 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790470007264306 us; error 58 us; skew 500 ppm
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:47.267617 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:47.268868 12715 webserver.cc:460] Webserver started at http://127.0.98.1:33151/ using document root <none> and password file <none>
I20260927 00:46:47.269615 12715 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:47.269708 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:47.269994 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:47.284794 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "0395d5b261ac4a39b7c7f5250946cae3"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "60bc104b128e9c71883237529fd83a8e"
server_key_iv: "94dd61cf4a8d055dd7b6dbd1b5700273"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.285495 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0395d5b261ac4a39b7c7f5250946cae3"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "60bc104b128e9c71883237529fd83a8e"
server_key_iv: "94dd61cf4a8d055dd7b6dbd1b5700273"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.290525 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:46:47.320433 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.323016 12715 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260927 00:46:47.323184 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "0395d5b261ac4a39b7c7f5250946cae3"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "60bc104b128e9c71883237529fd83a8e"
server_key_iv: "94dd61cf4a8d055dd7b6dbd1b5700273"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.323354 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:47.357993 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:47.361639 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:47.361913 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:47.362638 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:47.363654 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:47.363770 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.363969 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:47.364040 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.376441 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45521
I20260927 00:46:47.376644 12847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45521 every 8 connection(s)
I20260927 00:46:47.377859 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:47.381924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 12715
I20260927 00:46:47.382050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:47.382390 392 external_mini_cluster.cc:1428] Setting key 4a963a6138a4b65ba2181d78b5f210a4
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:47.395746 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.380152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37021 (local address 127.0.98.62:43725)
0927 00:46:47.380483 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:47.380493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:47.381321 (+ 828us) server_negotiation.cc:407] Connection header received
0927 00:46:47.382450 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:47.382454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:47.382516 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:47.382615 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:47.384194 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.385161 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.387218 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.387568 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.387646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.390894 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:47.390898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:47.390901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:47.390928 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:47.392876 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.393545 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.393548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.393550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.393645 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.394114 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.394118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.394120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.394526 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:47.394638 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:47.395414 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 00:46:47.395561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:47.396863 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.380662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.1:37021)
0927 00:46:47.381152 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:47.381211 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:47.382226 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:47.382940 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:47.382951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:47.383340 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:47.384011 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.384041 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.385349 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.385355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.387018 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.387076 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.387863 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.387867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.388091 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.388831 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:47.388852 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:47.390691 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:47.393035 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.393042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.393060 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.393420 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.393760 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.393763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.393765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.393982 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.394705 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:47.394711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:47.395087 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 00:46:47.395307 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 00:46:47.395736 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":150,"threads_started":1}
I20260927 00:46:47.398579 12848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:47.398849 12848 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:47.399422 12848 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:47.402245 12634 ts_manager.cc:194] Registered new tserver with Master: 0395d5b261ac4a39b7c7f5250946cae3 (127.0.98.1:45521)
I20260927 00:46:47.403877 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:47.417222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:46:47.546461 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:47.546746 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:47.546859 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:47.551028 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:47.551105 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:47.551250 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:47.556596 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:47.557686 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:47.558732 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:47.569728 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.569712 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.570215 12854 server_base.cc:1034] running on GCE node
W20260927 00:46:47.573222 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.573796 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:47.574615 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:47.582023 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790470007581981 us; error 51 us; skew 500 ppm
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:47.589475 12854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:47.590785 12854 webserver.cc:460] Webserver started at http://127.0.98.2:44467/ using document root <none> and password file <none>
I20260927 00:46:47.591625 12854 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:47.591786 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:47.592105 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:47.609999 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "b14ce513656c4ca9b56e3019cdb94088"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "379aa87f22425d38e7ddfdb8e54dc63e"
server_key_iv: "b4deea8d07a5cd283fa1a8e944d09364"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.610893 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b14ce513656c4ca9b56e3019cdb94088"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "379aa87f22425d38e7ddfdb8e54dc63e"
server_key_iv: "b4deea8d07a5cd283fa1a8e944d09364"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.616400 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:46:47.637612 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.639315 12854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:47.639443 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "b14ce513656c4ca9b56e3019cdb94088"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "379aa87f22425d38e7ddfdb8e54dc63e"
server_key_iv: "b4deea8d07a5cd283fa1a8e944d09364"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.639567 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:47.666498 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:47.670542 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:47.670804 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:47.671555 12854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:47.672662 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:47.672775 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.672868 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:47.672925 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.684962 12854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40431
I20260927 00:46:47.686344 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:47.687996 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40431 every 8 connection(s)
I20260927 00:46:47.689657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 12854
I20260927 00:46:47.689805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:47.690109 392 external_mini_cluster.cc:1428] Setting key 1db0825508687712cdf7d792cf67ec14
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:46:47.709614 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.691803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60241 (local address 127.0.98.62:43725)
0927 00:46:47.691945 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:47.691950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:47.692633 (+ 683us) server_negotiation.cc:407] Connection header received
0927 00:46:47.694083 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:47.694086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:47.694139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:47.694193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:47.695861 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.696688 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.701309 (+ 4621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.701622 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.701717 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.705223 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:47.705227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:47.705229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:47.705258 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:47.707172 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.707772 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.707775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.707777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.707863 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.708252 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.708255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.708257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.708470 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:47.708556 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:47.709291 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 00:46:47.709435 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:46:47.711416 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.690608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.2:60241)
0927 00:46:47.692472 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:47.692531 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:47.693837 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:47.694623 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:47.694632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:47.694995 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:47.695687 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.695708 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.696851 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.696854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.698293 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.698304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.701955 (+ 3651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.701960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.702170 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.703332 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:47.703355 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:47.705031 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:47.707288 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.707294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.707311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.707638 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.707961 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.707964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.707966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.708142 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.708608 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:47.708614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:47.708973 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 00:46:47.709906 (+ 933us) client_negotiation.cc:240] Negotiation successful
0927 00:46:47.710281 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:46:47.712833 12984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:47.713080 12984 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:47.713660 12984 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:47.714783 12634 ts_manager.cc:194] Registered new tserver with Master: b14ce513656c4ca9b56e3019cdb94088 (127.0.98.2:40431)
I20260927 00:46:47.715509 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:47.731202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:47.874157 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:47.874555 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:47.874653 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:47.880452 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:47.880612 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:47.880775 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:47.888624 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:47.890303 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:47.891712 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:46:47.907068 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.907533 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.907089 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.918145 12989 server_base.cc:1034] running on GCE node
I20260927 00:46:47.918800 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:47.919682 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:47.927075 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790470007926924 us; error 154 us; skew 500 ppm
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:47.931420 12989 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:47.932644 12989 webserver.cc:460] Webserver started at http://127.0.98.3:44255/ using document root <none> and password file <none>
I20260927 00:46:47.933519 12989 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:47.933665 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:47.934013 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:46:47 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:47.963021 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "d934a44cc2b745528a39f1bc37515daf"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "42f4a4d5fbd8f025b21736e668d58e15"
server_key_iv: "cbb6cdc6d265653483561cf8c3b8311c"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.967902 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d934a44cc2b745528a39f1bc37515daf"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "42f4a4d5fbd8f025b21736e668d58e15"
server_key_iv: "cbb6cdc6d265653483561cf8c3b8311c"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.974097 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 00:46:47.996567 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.998329 12989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:47.998466 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "d934a44cc2b745528a39f1bc37515daf"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "42f4a4d5fbd8f025b21736e668d58e15"
server_key_iv: "cbb6cdc6d265653483561cf8c3b8311c"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:47.998600 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.017614 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.021342 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.021523 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.022210 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:48.023175 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:48.023244 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.023305 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:48.023326 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.035629 12989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36083
I20260927 00:46:48.035852 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36083 every 8 connection(s)
I20260927 00:46:48.036650 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:48.042064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 12989
I20260927 00:46:48.042198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:48.042505 392 external_mini_cluster.cc:1428] Setting key 68de8effd1f2da0f983d1ccc42ffa43f
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:48.052539 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.038714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56829 (local address 127.0.98.62:43725)
0927 00:46:48.038838 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.038843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.039751 (+ 908us) server_negotiation.cc:407] Connection header received
0927 00:46:48.040743 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.040746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.040805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.040906 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.042125 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.042900 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.045047 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.045426 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.045490 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.049014 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.049018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.049021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.049051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.050535 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.051045 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.051048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.051050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.051127 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.051458 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.051462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.051463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.051640 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.051743 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.052237 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.052368 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:46:48.053462 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.039132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.3:56829)
0927 00:46:48.039594 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:48.039647 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:48.040533 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:48.041080 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:48.041087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:48.041348 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:48.041939 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.041961 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.043115 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.043119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.044899 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.044912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.045679 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.045682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.045887 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.047486 (+ 1599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:48.047503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:48.048846 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:48.050667 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.050672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.050688 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.050943 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.051213 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.051218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.051220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.051361 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.051741 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:48.051748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:48.052005 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 00:46:48.052198 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:46:48.052513 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":139,"threads_started":1}
I20260927 00:46:48.054564 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:48.054793 13118 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.055292 13118 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:48.056406 12634 ts_manager.cc:194] Registered new tserver with Master: d934a44cc2b745528a39f1bc37515daf (127.0.98.3:36083)
I20260927 00:46:48.056915 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56829
I20260927 00:46:48.068578 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:48.069393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 12715
I20260927 00:46:48.075974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 12854
I20260927 00:46:48.081461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 12989
I20260927 00:46:48.087569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 12602
I20260927 00:46:48.094165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43725
--webserver_interface=127.0.98.62
--webserver_port=37021
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43725
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:48.202207 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.202448 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:48.202519 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:48.206441 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:48.206491 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:48.206511 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:48.206559 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:48.206591 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:48.211788 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43725
--ranger_config_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:48.212812 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:48.213797 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:48.221942 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.221931 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.222040 13121 server_base.cc:1034] running on GCE node
W20260927 00:46:48.222280 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.222790 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:48.223814 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:48.224990 13121 hybrid_clock.cc:648] HybridClock initialized: now 1790470008224977 us; error 28 us; skew 500 ppm
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:48.227594 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:48.228395 13121 webserver.cc:460] Webserver started at http://127.0.98.62:37021/ using document root <none> and password file <none>
I20260927 00:46:48.228907 13121 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:48.228995 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:48.298188 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:48.300400 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.301685 13121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:48.301847 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "f7befba2315d43018d8694904b0316c2"
format_stamp: "Formatted at 2026-09-27 00:46:46 on dist-test-slave-l5sb"
server_key: "514b64f66b2d0a3e129923af861740b2"
server_key_iv: "0c040a63113be900355fc4528151e4cf"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:48.302410 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.322515 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.325523 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.325729 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.332779 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43725
I20260927 00:46:48.332787 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43725 every 8 connection(s)
I20260927 00:46:48.333827 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:48.337954 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:46:48.339511 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Bootstrap starting.
I20260927 00:46:48.340967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13121
I20260927 00:46:48.341449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45521
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43725
--webserver_port=33151
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:48.356029 13190 log.cc:824] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:48.362357 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:48.362740 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Bootstrap complete.
I20260927 00:46:48.366853 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:48.367226 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7befba2315d43018d8694904b0316c2, State: Initialized, Role: FOLLOWER
I20260927 00:46:48.367698 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:48.367805 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:48.367887 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:48.367980 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:46:48.369741 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:48.370107 13190 leader_election.cc:302] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7befba2315d43018d8694904b0316c2; no voters:
I20260927 00:46:48.370411 13190 leader_election.cc:288] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:46:48.370496 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:46:48.370687 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [term 2 LEADER]: Becoming Leader. State: Replica: f7befba2315d43018d8694904b0316c2, State: Running, Role: LEADER
I20260927 00:46:48.370961 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } }
I20260927 00:46:48.371589 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:48.371846 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7befba2315d43018d8694904b0316c2. Latest consensus state: current_term: 2 leader_uuid: "f7befba2315d43018d8694904b0316c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } } }
I20260927 00:46:48.371965 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:48.372212 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7befba2315d43018d8694904b0316c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7befba2315d43018d8694904b0316c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43725 } } }
I20260927 00:46:48.372310 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:48.372607 13198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:48.375892 13198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:48.376368 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:46:48.376576 13198 catalog_manager.cc:1259] Loaded cluster ID: c6e195b6ee37482f948ee284a3515e40
I20260927 00:46:48.376631 13198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:48.381557 13198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:48.382555 13198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Loaded TSK: 0
I20260927 00:46:48.383713 13198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:46:48.461509 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.461810 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:48.461891 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:48.466501 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:48.466588 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:48.466675 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:48.472426 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33151
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:48.473673 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:48.474819 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:48.486861 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.486907 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.487327 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.488276 13192 server_base.cc:1034] running on GCE node
I20260927 00:46:48.488763 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:48.489573 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:48.490777 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790470008490743 us; error 47 us; skew 500 ppm
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:48.494334 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:48.495419 13192 webserver.cc:460] Webserver started at http://127.0.98.1:33151/ using document root <none> and password file <none>
I20260927 00:46:48.496095 13192 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:48.496196 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:48.518208 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 00:46:48.522576 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.524140 13192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:46:48.524272 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "0395d5b261ac4a39b7c7f5250946cae3"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "60bc104b128e9c71883237529fd83a8e"
server_key_iv: "94dd61cf4a8d055dd7b6dbd1b5700273"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:48.524996 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.546007 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.549682 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.549909 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.550597 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:48.551601 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:48.551657 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.551707 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:48.551735 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.562731 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45521
I20260927 00:46:48.562780 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45521 every 8 connection(s)
I20260927 00:46:48.564033 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:48.570160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13192
I20260927 00:46:48.570640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40431
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43725
--webserver_port=44467
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:48.584630 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.566403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.1:37653)
0927 00:46:48.567181 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:48.567261 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:48.568351 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:48.570261 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:48.570270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:48.570607 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:48.571242 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.571263 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.573167 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.573170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.574570 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.574580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.575738 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.575744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.575939 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.576639 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:48.576661 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:48.578419 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:48.580950 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.580956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.580970 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.581309 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.581718 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.581721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.581723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.581923 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.582728 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:48.582734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:48.583063 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 00:46:48.583261 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:46:48.583591 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":143,"threads_started":1}
I20260927 00:46:48.586411 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.566348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37653 (local address 127.0.98.62:43725)
0927 00:46:48.566863 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.566886 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.567410 (+ 524us) server_negotiation.cc:407] Connection header received
0927 00:46:48.568854 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.568870 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.569295 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.569625 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.571442 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.572993 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.574705 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.575374 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.575523 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.578641 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.578648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.578667 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.578700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.580810 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.581447 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.581454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.581461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.581604 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.582051 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.582055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.582056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.582542 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.582674 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.585201 (+ 2527us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.585495 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":142,"threads_started":1}
I20260927 00:46:48.588994 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:48.589340 13352 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.589967 13352 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:48.592206 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:46:48.592332 13352 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.592510 13352 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:48.593302 13352 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.593464 13352 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:48.594200 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:46:48.715545 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.715842 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:48.715921 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:48.720118 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:48.720201 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:48.720286 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:48.726128 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44467
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:48.727226 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:48.728255 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:48.737949 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.739110 13356 server_base.cc:1034] running on GCE node
W20260927 00:46:48.739364 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.741937 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.742432 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:48.743175 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:48.744355 13356 hybrid_clock.cc:648] HybridClock initialized: now 1790470008744320 us; error 49 us; skew 500 ppm
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:48.747308 13356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:48.748291 13356 webserver.cc:460] Webserver started at http://127.0.98.2:44467/ using document root <none> and password file <none>
I20260927 00:46:48.748862 13356 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:48.748966 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:48.769539 13356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:46:48.777403 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.779500 13356 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260927 00:46:48.779642 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "b14ce513656c4ca9b56e3019cdb94088"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "379aa87f22425d38e7ddfdb8e54dc63e"
server_key_iv: "b4deea8d07a5cd283fa1a8e944d09364"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:48.780427 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.816787 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.820860 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.821070 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.821842 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:48.822899 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:48.822973 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.823029 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:48.823065 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.834792 13356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40431
I20260927 00:46:48.834880 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40431 every 8 connection(s)
I20260927 00:46:48.836107 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:48.843374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13356
I20260927 00:46:48.843860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36083
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43725
--webserver_port=44255
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:46:48 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:48.856740 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.838798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33295 (local address 127.0.98.62:43725)
0927 00:46:48.838929 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.838933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.842296 (+ 3363us) server_negotiation.cc:407] Connection header received
0927 00:46:48.843640 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.843644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.843709 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.843766 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.845433 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.846267 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.847992 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.848347 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.848459 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.851655 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.851659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.851662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.851696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.853559 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.854712 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.854716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.854718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.854817 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.855272 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.855276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.855278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.855542 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.855636 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.856394 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.856558 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:46:48.858778 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.839304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.2:33295)
0927 00:46:48.842116 (+ 2812us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:48.842176 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:48.843349 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:48.844180 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:48.844189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:48.844552 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:48.845243 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.845266 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.846378 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.846381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.847842 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.847854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.848655 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.848662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.848865 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.849623 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:48.849646 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:48.851427 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:48.853919 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.853925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.853944 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.854530 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.854945 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.854948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.854950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.855146 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.855710 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:48.855716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:48.856083 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 00:46:48.857257 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0927 00:46:48.857634 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":146,"threads_started":1}
I20260927 00:46:48.860222 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:48.860489 13486 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.861096 13486 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:48.862813 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:46:48.862927 13486 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.863078 13486 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:48.863875 13486 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.864028 13486 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:48.864734 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:46:48.999607 13489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.999956 13489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:49.000033 13489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:49.005904 13489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:49.006045 13489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:49.006163 13489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:49.011746 13489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44255
--tserver_master_addrs=127.0.98.62:43725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:49.012789 13489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:49.013916 13489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:49.024855 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:49.024845 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:49.025171 13489 server_base.cc:1034] running on GCE node
W20260927 00:46:49.025269 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:49.025673 13489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:49.026309 13489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:49.027530 13489 hybrid_clock.cc:648] HybridClock initialized: now 1790470009027498 us; error 48 us; skew 500 ppm
Sep 27 00:46:49 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:49.031383 13489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:49.032358 13489 webserver.cc:460] Webserver started at http://127.0.98.3:44255/ using document root <none> and password file <none>
I20260927 00:46:49.033037 13489 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:49.033135 13489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:46:49 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:49.069689 13489 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.000s
I20260927 00:46:49.072978 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:49.075749 13489 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 00:46:49.075908 13489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "d934a44cc2b745528a39f1bc37515daf"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-l5sb"
server_key: "42f4a4d5fbd8f025b21736e668d58e15"
server_key_iv: "cbb6cdc6d265653483561cf8c3b8311c"
server_key_version: "kuduclusterkey@0"
I20260927 00:46:49.076678 13489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:49.117913 13489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:49.121694 13489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:49.121984 13489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:49.122750 13489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:49.123905 13489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:49.123992 13489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:49.124085 13489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:49.124131 13489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:49.135135 13489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36083
I20260927 00:46:49.135396 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36083 every 8 connection(s)
I20260927 00:46:49.136526 13489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:49.141366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13489
Sep 27 00:46:49 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:49.153314 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:49.142094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.98.62:43725)
0927 00:46:49.143110 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:49.143114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:49.143393 (+ 279us) server_negotiation.cc:407] Connection header received
0927 00:46:49.143594 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:49.143597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:49.143656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:49.143720 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:49.144726 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.145272 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.146505 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.146881 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.146986 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:49.148707 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:49.148713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:49.148719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:49.148743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:49.151437 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:49.151941 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:49.151945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:49.151947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:49.152038 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:49.152469 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:49.152471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:49.152473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:49.152661 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:49.152872 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:49.153006 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:46:49.153126 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":943,"spinlock_wait_cycles":1672576,"thread_start_us":93,"threads_started":1}
I20260927 00:46:49.155579 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:49.140502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48979 (local address 127.0.98.62:43725)
0927 00:46:49.141928 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:49.141935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:49.141953 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:46:49.142743 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:49.142747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:49.142804 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:49.142860 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:49.144620 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.145428 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.146899 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.147228 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.147344 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:49.150339 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:49.150342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:49.150345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:49.151391 (+ 1046us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:49.153085 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:49.153760 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:49.153763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:49.153796 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:49.153891 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:49.154339 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:49.154344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:49.154346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:49.154558 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:49.154645 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:49.155383 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 00:46:49.155473 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1313,"spinlock_wait_cycles":5504}
I20260927 00:46:49.156744 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:49.139374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43725 (local address 127.0.98.3:48979)
0927 00:46:49.140988 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:49.141043 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:49.142161 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:49.143310 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:49.143322 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:49.143709 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:49.144443 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:49.144463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.145564 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.145568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:49.146770 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:49.146779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:49.147403 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:49.147408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:49.147618 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:49.148256 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:49.148277 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:49.150151 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:49.153244 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:49.153251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:49.153272 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:49.153634 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:49.154016 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:49.154019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:49.154021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:49.154217 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:49.154721 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:49.154728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:49.155091 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 00:46:49.155334 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 00:46:49.155682 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1383,"thread_start_us":128,"threads_started":1}
I20260927 00:46:49.157919 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43725
I20260927 00:46:49.158193 13620 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:49.158792 13620 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:49.160324 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:46:49.160466 13620 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:49.160656 13620 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:49.161501 13620 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:49.161684 13620 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
W20260927 00:46:49.162448 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:46:49 dist-test-slave-l5sb krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:49.594411 13352 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:49.594736 13352 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:49.595852 13154 ts_manager.cc:194] Registered new tserver with Master: 0395d5b261ac4a39b7c7f5250946cae3 (127.0.98.1:45521)
I20260927 00:46:49.597462 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37653
I20260927 00:46:49.864975 13486 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:49.865345 13486 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:49.866166 13154 ts_manager.cc:194] Registered new tserver with Master: b14ce513656c4ca9b56e3019cdb94088 (127.0.98.2:40431)
I20260927 00:46:49.866777 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33295
Sep 27 00:46:49 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:50.162705 13620 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:50.163017 13620 heartbeater.cc:511] Master 127.0.98.62:43725 requested a full tablet report, sending...
I20260927 00:46:50.163834 13154 ts_manager.cc:194] Registered new tserver with Master: d934a44cc2b745528a39f1bc37515daf (127.0.98.3:36083)
I20260927 00:46:50.164491 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48979
I20260927 00:46:50.168692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:50.183768 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.172347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.0.98.62:43725)
0927 00:46:50.172670 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.172674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.172743 (+ 69us) server_negotiation.cc:407] Connection header received
0927 00:46:50.172895 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.172898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.172947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.173107 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:50.174535 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.175377 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.176533 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.176759 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.176867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.178561 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:50.178564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:50.178567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:50.178594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:50.180364 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.181868 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.181875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.181879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.181986 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.182318 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.182321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.182323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.182548 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:50.182675 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.183450 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.183579 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
Sep 27 00:46:50 dist-test-slave-l5sb krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:46:50.187934 13154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:50.193397 13154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:50.243765 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.233489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.98.1:45521)
0927 00:46:50.234315 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.234323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.236382 (+ 2059us) server_negotiation.cc:407] Connection header received
0927 00:46:50.236684 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.236693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.236922 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.237079 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:50.237641 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.239664 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.241832 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.242450 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.242556 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.242720 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.243499 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.243605 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":83,"threads_started":1}
I20260927 00:46:50.244509 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.232584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45521 (local address 127.0.0.1:37206)
0927 00:46:50.236238 (+ 3654us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.236296 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.236493 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.237335 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.237341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.237367 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:50.237501 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.237508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.239809 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.239812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.241478 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.241488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.243013 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.243022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.243265 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.243317 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.244232 (+ 915us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.244368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"thread_start_us":138,"threads_started":1}
I20260927 00:46:50.248791 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded (DEFAULT_TABLE table=test-table [id=7c1d63e3f5ef4ed5a02dc0abde1b3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:50.250168 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:50.260380 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.233977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40431 (local address 127.0.0.1:33094)
0927 00:46:50.238978 (+ 5001us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.238995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.239084 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.240183 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.240186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.240203 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:50.240292 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.240297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.244038 (+ 3741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.244044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.245243 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.245254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.259908 (+ 14654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.259912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.260031 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.260059 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.260137 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.260226 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4936,"thread_start_us":87,"threads_started":1}
I20260927 00:46:50.261004 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.234121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.98.2:40431)
0927 00:46:50.239554 (+ 5433us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.239562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.239587 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:50.239658 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.239663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.239856 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.239992 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:50.242677 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.243892 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.253932 (+ 10040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.254872 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.254992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.255150 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.260741 (+ 5591us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.260863 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5315,"thread_start_us":125,"threads_started":1}
I20260927 00:46:50.269557 13419 tablet_service.cc:1467] Processing CreateTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded (DEFAULT_TABLE table=test-table [id=7c1d63e3f5ef4ed5a02dc0abde1b3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:50.271016 13419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:50.285077 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.234706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36083 (local address 127.0.0.1:40808)
0927 00:46:50.239244 (+ 4538us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.239255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.239316 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.246222 (+ 6906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.246225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.246242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:50.251375 (+ 5133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.251388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.282212 (+ 30824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.282217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.283449 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.283463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.284640 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.284644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.284752 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.284783 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.284832 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.284913 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4488,"thread_start_us":84,"threads_started":1}
I20260927 00:46:50.286202 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.234862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.0.98.3:36083)
0927 00:46:50.245557 (+ 10695us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.245564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.245587 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:46:50.245661 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.245666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.245919 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.246067 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:50.261434 (+ 15367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.281991 (+ 20557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.283635 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.284465 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.285618 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.285906 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.285961 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.286062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10571,"thread_start_us":93,"threads_started":1}
I20260927 00:46:50.289796 13535 tablet_service.cc:1467] Processing CreateTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded (DEFAULT_TABLE table=test-table [id=7c1d63e3f5ef4ed5a02dc0abde1b3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:50.290930 13535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:50.299106 13656 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Bootstrap starting.
I20260927 00:46:50.301512 13655 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: Bootstrap starting.
I20260927 00:46:50.304131 13655 tablet_bootstrap.cc:654] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:50.304571 13657 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Bootstrap starting.
I20260927 00:46:50.305272 13656 tablet_bootstrap.cc:654] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:50.306660 13657 tablet_bootstrap.cc:654] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:50.307577 13657 log.cc:824] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:50.307588 13656 log.cc:824] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:50.308595 13655 log.cc:824] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:50.309863 13657 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: No bootstrap required, opened a new log
I20260927 00:46:50.310071 13657 ts_tablet_manager.cc:1397] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:46:50.310618 13656 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: No bootstrap required, opened a new log
I20260927 00:46:50.310781 13656 ts_tablet_manager.cc:1397] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260927 00:46:50.314488 13657 raft_consensus.cc:348] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.314792 13657 raft_consensus.cc:374] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:50.314857 13657 raft_consensus.cc:729] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d934a44cc2b745528a39f1bc37515daf, State: Initialized, Role: FOLLOWER
I20260927 00:46:50.314692 13656 raft_consensus.cc:348] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.314945 13656 raft_consensus.cc:374] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:50.314988 13656 raft_consensus.cc:729] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14ce513656c4ca9b56e3019cdb94088, State: Initialized, Role: FOLLOWER
I20260927 00:46:50.315475 13657 consensus_queue.cc:260] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.315737 13656 consensus_queue.cc:260] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.316618 13657 ts_tablet_manager.cc:1428] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:46:50.317006 13620 heartbeater.cc:503] Master 127.0.98.62:43725 was elected leader, sending a full tablet report...
I20260927 00:46:50.321285 13656 ts_tablet_manager.cc:1428] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260927 00:46:50.325393 13655 tablet_bootstrap.cc:492] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: No bootstrap required, opened a new log
I20260927 00:46:50.325697 13655 ts_tablet_manager.cc:1397] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260927 00:46:50.329020 13486 heartbeater.cc:503] Master 127.0.98.62:43725 was elected leader, sending a full tablet report...
I20260927 00:46:50.332736 13655 raft_consensus.cc:348] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.333148 13655 raft_consensus.cc:374] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:50.333246 13655 raft_consensus.cc:729] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0395d5b261ac4a39b7c7f5250946cae3, State: Initialized, Role: FOLLOWER
I20260927 00:46:50.333812 13655 consensus_queue.cc:260] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.335057 13655 ts_tablet_manager.cc:1428] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 00:46:50.335445 13352 heartbeater.cc:503] Master 127.0.98.62:43725 was elected leader, sending a full tablet report...
W20260927 00:46:50.338205 13487 tablet.cc:2369] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:46:50.391762 13621 tablet.cc:2369] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:50.406961 13662 raft_consensus.cc:482] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:50.407207 13662 raft_consensus.cc:504] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.408576 13662 leader_election.cc:288] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0395d5b261ac4a39b7c7f5250946cae3 (127.0.98.1:45521), d934a44cc2b745528a39f1bc37515daf (127.0.98.3:36083)
I20260927 00:46:50.414330 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.409243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57729 (local address 127.0.98.1:45521)
0927 00:46:50.409368 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.409372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.409671 (+ 299us) server_negotiation.cc:407] Connection header received
0927 00:46:50.409843 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.409857 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.409914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.409996 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:50.410439 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.411254 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.412865 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.413563 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.413644 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.413711 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.414117 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.414195 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:50.414340 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.409350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56375 (local address 127.0.98.3:36083)
0927 00:46:50.409464 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.409468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.410068 (+ 600us) server_negotiation.cc:407] Connection header received
0927 00:46:50.410258 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.410261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.410316 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.410366 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:50.410780 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.411550 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.413028 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.413655 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.413731 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.413814 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.414134 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.414209 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:46:50.414801 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.408969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36083 (local address 127.0.98.2:56375)
0927 00:46:50.409982 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.409995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.410137 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.410502 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.410506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.410522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:50.410645 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.410652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.411704 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.411708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.412889 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.412900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.413903 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.413907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.414006 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.414026 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.414539 (+ 513us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.414615 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":109,"threads_started":1}
I20260927 00:46:50.415323 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.408827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45521 (local address 127.0.98.2:57729)
0927 00:46:50.409542 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.409568 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.409710 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.410135 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.410138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.410158 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:50.410300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.410307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.411398 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.411401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.412714 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.412725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.413830 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.413834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.413957 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.413980 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.414959 (+ 979us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.415183 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":13,"thread_start_us":108,"threads_started":1}
I20260927 00:46:50.415589 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d21e42cd5ad4bc89ea2744bdd7a5ded" candidate_uuid: "b14ce513656c4ca9b56e3019cdb94088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d934a44cc2b745528a39f1bc37515daf" is_pre_election: true
I20260927 00:46:50.415953 13567 raft_consensus.cc:2454] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14ce513656c4ca9b56e3019cdb94088 in term 0.
I20260927 00:46:50.415937 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d21e42cd5ad4bc89ea2744bdd7a5ded" candidate_uuid: "b14ce513656c4ca9b56e3019cdb94088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0395d5b261ac4a39b7c7f5250946cae3" is_pre_election: true
I20260927 00:46:50.416214 13305 raft_consensus.cc:2454] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14ce513656c4ca9b56e3019cdb94088 in term 0.
I20260927 00:46:50.416493 13376 leader_election.cc:302] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14ce513656c4ca9b56e3019cdb94088, d934a44cc2b745528a39f1bc37515daf; no voters:
I20260927 00:46:50.416795 13662 raft_consensus.cc:2781] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:50.416888 13662 raft_consensus.cc:482] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:50.416935 13662 raft_consensus.cc:3037] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:50.418376 13662 raft_consensus.cc:504] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.418756 13662 leader_election.cc:288] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [CANDIDATE]: Term 1 election: Requested vote from peers 0395d5b261ac4a39b7c7f5250946cae3 (127.0.98.1:45521), d934a44cc2b745528a39f1bc37515daf (127.0.98.3:36083)
I20260927 00:46:50.419137 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d21e42cd5ad4bc89ea2744bdd7a5ded" candidate_uuid: "b14ce513656c4ca9b56e3019cdb94088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0395d5b261ac4a39b7c7f5250946cae3"
I20260927 00:46:50.419263 13305 raft_consensus.cc:3037] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:50.419368 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d21e42cd5ad4bc89ea2744bdd7a5ded" candidate_uuid: "b14ce513656c4ca9b56e3019cdb94088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d934a44cc2b745528a39f1bc37515daf"
I20260927 00:46:50.419490 13567 raft_consensus.cc:3037] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:50.420583 13305 raft_consensus.cc:2454] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14ce513656c4ca9b56e3019cdb94088 in term 1.
I20260927 00:46:50.420744 13567 raft_consensus.cc:2454] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14ce513656c4ca9b56e3019cdb94088 in term 1.
I20260927 00:46:50.421185 13375 leader_election.cc:302] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0395d5b261ac4a39b7c7f5250946cae3, b14ce513656c4ca9b56e3019cdb94088; no voters:
I20260927 00:46:50.421595 13662 raft_consensus.cc:2781] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:50.421841 13662 raft_consensus.cc:686] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 1 LEADER]: Becoming Leader. State: Replica: b14ce513656c4ca9b56e3019cdb94088, State: Running, Role: LEADER
I20260927 00:46:50.422215 13662 consensus_queue.cc:237] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } }
I20260927 00:46:50.425984 13154 catalog_manager.cc:5495] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 reported cstate change: term changed from 0 to 1, leader changed from <none> to b14ce513656c4ca9b56e3019cdb94088 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b14ce513656c4ca9b56e3019cdb94088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b14ce513656c4ca9b56e3019cdb94088" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 } health_report { overall_health: UNKNOWN } } }
I20260927 00:46:50.444635 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.439335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.98.2:40431)
0927 00:46:50.439613 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.439617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.439630 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:46:50.439709 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.439712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.439778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.439879 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:50.440260 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.441078 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.442399 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.442745 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.442848 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.443208 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:50.443786 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:50.443908 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.444318 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.444382 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260927 00:46:50.452714 13567 raft_consensus.cc:1261] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 1 FOLLOWER]: Refusing update from remote peer b14ce513656c4ca9b56e3019cdb94088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:50.452788 13305 raft_consensus.cc:1261] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 1 FOLLOWER]: Refusing update from remote peer b14ce513656c4ca9b56e3019cdb94088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:50.453606 13666 consensus_queue.cc:1035] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d934a44cc2b745528a39f1bc37515daf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:50.453794 13662 consensus_queue.cc:1035] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0395d5b261ac4a39b7c7f5250946cae3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:50.463364 13678 mvcc.cc:205] Tried to move back new op lower bound from 7333765162806312960 to 7333765162692579328. Current Snapshot: MvccSnapshot[applied={T|T < 7333765162806312960}]
I20260927 00:46:50.465799 13677 mvcc.cc:205] Tried to move back new op lower bound from 7333765162806312960 to 7333765162692579328. Current Snapshot: MvccSnapshot[applied={T|T < 7333765162806312960}]
I20260927 00:46:50.474072 13154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:50.474258 13154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:50.476732 13154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7befba2315d43018d8694904b0316c2: Sending DeleteTablet for 3 replicas of tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded
I20260927 00:46:50.477600 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37206
I20260927 00:46:50.477667 13535 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40808
I20260927 00:46:50.478138 13686 tablet_replica.cc:330] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3: stopping tablet replica
I20260927 00:46:50.478129 13419 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33094
I20260927 00:46:50.478574 13686 raft_consensus.cc:2229] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:50.478861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13192
I20260927 00:46:50.478943 13686 raft_consensus.cc:2258] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P 0395d5b261ac4a39b7c7f5250946cae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:50.479301 13688 tablet_replica.cc:330] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: stopping tablet replica
I20260927 00:46:50.479758 13688 raft_consensus.cc:2229] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:50.480156 13688 raft_consensus.cc:2258] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:50.481525 13688 ts_tablet_manager.cc:1905] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:50.484157 13688 ts_tablet_manager.cc:1918] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:50.484268 13688 log.cc:1196] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/9d21e42cd5ad4bc89ea2744bdd7a5ded
I20260927 00:46:50.484581 13688 ts_tablet_manager.cc:1939] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P b14ce513656c4ca9b56e3019cdb94088: Deleting consensus metadata
I20260927 00:46:50.485766 13138 catalog_manager.cc:4841] TS b14ce513656c4ca9b56e3019cdb94088 (127.0.98.2:40431): tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded (table test-table [id=7c1d63e3f5ef4ed5a02dc0abde1b3b01]) successfully deleted
I20260927 00:46:50.488667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13356
W20260927 00:46:50.488631 13140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45521 (error 108)
I20260927 00:46:50.488741 13687 tablet_replica.cc:330] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: stopping tablet replica
I20260927 00:46:50.490315 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.489957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45521 (local address 127.0.0.1:37210)
0927 00:46:50.490116 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.490203 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:46:50.490466 13140 catalog_manager.cc:4584] TS 0395d5b261ac4a39b7c7f5250946cae3 (127.0.98.1:45521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d21e42cd5ad4bc89ea2744bdd7a5ded: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45521: connect: Connection refused (error 111)
I20260927 00:46:50.490630 13687 raft_consensus.cc:2229] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:50.490942 13687 raft_consensus.cc:2258] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:50.492240 13687 ts_tablet_manager.cc:1905] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:50.494554 13687 ts_tablet_manager.cc:1918] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:50.494680 13687 log.cc:1196] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/9d21e42cd5ad4bc89ea2744bdd7a5ded
I20260927 00:46:50.494807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13489
I20260927 00:46:50.495023 13687 ts_tablet_manager.cc:1939] T 9d21e42cd5ad4bc89ea2744bdd7a5ded P d934a44cc2b745528a39f1bc37515daf: Deleting consensus metadata
I20260927 00:46:50.500811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13121
I20260927 00:46:50.507140 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:46:50.842070 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:46:50.842178 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:46:51.177158 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:46:51.177 UTC [12188] LOG: received smart shutdown request
2026-09-27 00:46:51.181 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-09-27 00:46:51.181 UTC [12192] LOG: shutting down
2026-09-27 00:46:51.351 UTC [12188] LOG: database system is shut down
2026-09-27T00:46:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48119 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:51 dist-test-slave-l5sb krb5kdc[13698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:51 dist-test-slave-l5sb krb5kdc[13698](info): set up 2 sockets
Sep 27 00:46:51 dist-test-slave-l5sb krb5kdc[13698](info): commencing operation
krb5kdc: starting...
W20260927 00:46:52.513948 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:52 dist-test-slave-l5sb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:52.632664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35339
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:52.758502 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:52.758805 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:52.758877 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:52.762816 13711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:52.762903 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:52.762957 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:52.763029 13711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:52.763098 13711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:52.768177 13711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:52.769173 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:52.770177 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:52.780540 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:52.780540 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:52.780838 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:52.781100 13711 server_base.cc:1034] running on GCE node
I20260927 00:46:52.781723 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:52.782832 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:52.784026 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790470012784017 us; error 30 us; skew 500 ppm
Sep 27 00:46:52 dist-test-slave-l5sb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:52.786780 13711 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:52.787647 13711 webserver.cc:460] Webserver started at http://127.0.98.62:40121/ using document root <none> and password file <none>
I20260927 00:46:52.788194 13711 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:52.788285 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:52.788512 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:52.790623 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "156873d04adb4056bee5edecb68276bb"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "15329f5f39acdfcced75a50e8752b527"
server_key_iv: "6af5f7d728ec3accc5ea0c752f9a1896"
server_key_version: "encryptionkey@0"
I20260927 00:46:52.791182 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "156873d04adb4056bee5edecb68276bb"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "15329f5f39acdfcced75a50e8752b527"
server_key_iv: "6af5f7d728ec3accc5ea0c752f9a1896"
server_key_version: "encryptionkey@0"
I20260927 00:46:52.794872 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:46:52.797237 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:52.798468 13711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:46:52.798605 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "156873d04adb4056bee5edecb68276bb"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "15329f5f39acdfcced75a50e8752b527"
server_key_iv: "6af5f7d728ec3accc5ea0c752f9a1896"
server_key_version: "encryptionkey@0"
I20260927 00:46:52.798712 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:52.812865 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:52.815747 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:52.815938 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:52.822619 13711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35339
I20260927 00:46:52.822701 13778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35339 every 8 connection(s)
I20260927 00:46:52.823530 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:52.826208 13779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:52.830008 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:52.830551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13711
I20260927 00:46:52.830685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:52.830950 392 external_mini_cluster.cc:1428] Setting key 3f18b5751386f5e6c75f8f24ad789f0d
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:46:52 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:52.846637 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.832298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35339 (local address 127.0.0.1:56228)
0927 00:46:52.832808 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.832850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:52.833731 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:52.834435 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:52.834440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:52.834881 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:52.835106 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:52.835118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:52.836213 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:52.836218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:52.836935 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:52.836957 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:52.837608 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:52.837612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:52.837710 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:52.838357 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:52.838373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:52.841476 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:52.843773 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:52.843780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:52.843788 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:52.843997 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:52.844402 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:52.844404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:52.844405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:52.844509 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:52.845268 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:52.845273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:52.845391 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 00:46:52.845524 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:46:52.845812 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":163,"threads_started":1}
I20260927 00:46:52.846642 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.833249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.0.98.62:35339)
0927 00:46:52.833543 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:52.833552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:52.833600 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:46:52.834074 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:52.834078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:52.834254 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:52.834435 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:52.835251 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:52.836074 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:52.837093 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:52.837466 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:52.837639 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:52.841638 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:52.841643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:52.841652 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:52.841673 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:52.843653 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:52.844129 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:52.844138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:52.844143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:52.844294 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:52.844602 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:52.844607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:52.844611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:52.845150 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:52.845302 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:52.845646 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 00:46:52.845878 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":192,"spinlock_wait_cycles":896,"thread_start_us":87,"threads_started":1}
I20260927 00:46:52.849285 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:52.850160 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.849833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50164)
0927 00:46:52.849977 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.850102 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:46:52.850407 13729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
I20260927 00:46:52.851219 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.850991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50178)
0927 00:46:52.851107 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.851149 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:46:52.851378 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37717: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:46:52.856923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37717
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:52.900612 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } attempt: 1
I20260927 00:46:52.901811 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.901253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50184)
0927 00:46:52.901618 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.901688 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 00:46:52.902623 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.902355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50196)
0927 00:46:52.902461 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.902557 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:46:52.902778 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37717: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
W20260927 00:46:52.965574 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:52.965950 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:52.966028 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:52.970047 13786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:52.970148 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:52.970186 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:52.970223 13786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:52.970256 13786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:52.970952 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } attempt: 2
I20260927 00:46:52.971757 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.971466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50212)
0927 00:46:52.971617 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.971684 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:46:52.972448 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:52.972233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50216)
0927 00:46:52.972349 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:52.972400 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:46:52.972579 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37717: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37717: connect: Connection refused (error 111)
I20260927 00:46:52.976035 13786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:52.977104 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:52.978147 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:46:52.986423 13786 server_base.cc:1034] running on GCE node
W20260927 00:46:52.986399 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:52.986440 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:52.986609 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:52.987123 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:52.988243 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:52.989423 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790470012989389 us; error 46 us; skew 500 ppm
Sep 27 00:46:52 dist-test-slave-l5sb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:52.992125 13786 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:46:52.992921 13786 webserver.cc:460] Webserver started at http://127.0.98.61:43879/ using document root <none> and password file <none>
I20260927 00:46:52.993407 13786 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:52.993460 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:52.993621 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:52.995661 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data/instance:
uuid: "f0d1e679b1a14bc09f03294e88d9cc04"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "70935a55d44e840de17a5305c75aed03"
server_key_iv: "79ce81164f99f7a3349ecbf80715098e"
server_key_version: "encryptionkey@0"
I20260927 00:46:52.996138 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal/instance:
uuid: "f0d1e679b1a14bc09f03294e88d9cc04"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "70935a55d44e840de17a5305c75aed03"
server_key_iv: "79ce81164f99f7a3349ecbf80715098e"
server_key_version: "encryptionkey@0"
I20260927 00:46:52.999599 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:46:53.001896 13801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.003078 13786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:46:53.003201 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "f0d1e679b1a14bc09f03294e88d9cc04"
format_stamp: "Formatted at 2026-09-27 00:46:52 on dist-test-slave-l5sb"
server_key: "70935a55d44e840de17a5305c75aed03"
server_key_iv: "79ce81164f99f7a3349ecbf80715098e"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.003302 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:53.020308 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:53.023535 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:53.023741 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:53.030617 13786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37717
I20260927 00:46:53.030653 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37717 every 8 connection(s)
I20260927 00:46:53.031513 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:46:53.032424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13786
I20260927 00:46:53.032552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-1/wal/instance
I20260927 00:46:53.032812 392 external_mini_cluster.cc:1428] Setting key 5ab9707ffe64ae27cb50792fed70c729
I20260927 00:46:53.034521 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:53.039094 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:53.054065 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.040938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.98.62:35339)
0927 00:46:53.041079 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.041083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.042005 (+ 922us) server_negotiation.cc:407] Connection header received
0927 00:46:53.043008 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.043011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.043062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.043145 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.044710 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.045235 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.047057 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.047291 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.047401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.049807 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.049811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.049814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.049847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.051938 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.052456 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.052460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.052462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.052542 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.052893 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.052898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.052900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.053086 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.053198 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.053721 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.053877 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:53.054884 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.041321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35339 (local address 127.0.0.1:56242)
0927 00:46:53.041859 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.041902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.042820 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.043385 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.043395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.043823 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.044541 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.044563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.045367 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.045373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.046918 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.046932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.047447 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.047451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.047623 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.048205 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.048221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.049608 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.052062 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.052068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.052084 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.052346 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.052647 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.052650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.052652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.052794 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.053198 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.053206 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.053473 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.053680 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.054043 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260927 00:46:53.056318 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:46:53.060233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42967
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:46:53.063716 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } attempt: 3
I20260927 00:46:53.065384 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.056798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50222)
0927 00:46:53.057007 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.057018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.057077 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.057631 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.057633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.057840 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.057963 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.057973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.058942 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.058945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.059544 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.059553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.060133 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.060137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.060264 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.060723 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.060734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.061936 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.063666 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.063672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.063675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.063895 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.064207 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.064209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.064211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.064291 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.064874 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.064879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.064973 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.065082 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.065219 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3}
I20260927 00:46:53.065876 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.056983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.0.98.61:37717)
0927 00:46:53.057298 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.057306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.057323 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:46:53.057354 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.057358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.057506 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.057647 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.058083 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.058829 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.059667 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.060020 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.060190 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.062059 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.062076 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.062086 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.062105 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.063553 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.064007 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.064019 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.064021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.064116 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.064395 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.064399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.064400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.064762 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.064909 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.065198 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.065369 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260927 00:46:53.067863 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:53.068553 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.068318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33958)
0927 00:46:53.068426 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.068501 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:46:53.069196 13804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:46:53.070134 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.069871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33962)
0927 00:46:53.069992 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.070055 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:46:53.070541 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
I20260927 00:46:53.073153 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.064243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.0.98.61:37717)
0927 00:46:53.064536 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.064542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.064552 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:46:53.064576 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.064578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.064609 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.064713 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.065208 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.066011 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.066878 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.067203 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.067296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.069900 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.069904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.069906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.069929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.071524 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.072031 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.072033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.072034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.072105 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.072420 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.072422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.072422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.072584 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.072658 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.072936 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.073025 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260927 00:46:53.073189 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.064171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50228)
0927 00:46:53.064319 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.064355 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.064440 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.064739 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.064743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.064944 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.065075 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.065084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.066156 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.066160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.066712 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.066721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.067339 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.067345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.067502 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.068037 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.068059 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.069726 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.071640 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.071646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.071651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.071930 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.072208 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.072213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.072218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.072325 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.072674 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.072677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.072783 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.072890 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.073028 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 00:46:53.074075 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:53.074771 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.074521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33970)
0927 00:46:53.074644 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.074703 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:46:53.075428 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.075189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33972)
0927 00:46:53.075325 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.075373 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 00:46:53.075554 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
I20260927 00:46:53.106719 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 1
I20260927 00:46:53.107643 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.107306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33984)
0927 00:46:53.107471 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.107542 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:46:53.108335 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.108107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:33998)
0927 00:46:53.108222 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.108272 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:46:53.108464 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
I20260927 00:46:53.119827 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 1
I20260927 00:46:53.120529 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.120257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34002)
0927 00:46:53.120392 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.120452 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:46:53.121236 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.121038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34010)
0927 00:46:53.121139 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.121183 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:46:53.121431 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
W20260927 00:46:53.174397 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:53.174667 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:53.174707 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:53.178865 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:53.178927 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:53.178946 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:53.178964 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:53.179016 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:53.183615 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 2
I20260927 00:46:53.184480 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.184155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34018)
0927 00:46:53.184323 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.184391 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:46:53.184882 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35339,127.0.98.61:37717,127.0.98.60:42967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:53.185359 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.185133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34022)
0927 00:46:53.185245 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.185296 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:46:53.185491 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
I20260927 00:46:53.186182 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:53.187144 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:46:53.189595 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 2
I20260927 00:46:53.190312 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.190070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34036)
0927 00:46:53.190195 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.190247 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:46:53.191063 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.190836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34042)
0927 00:46:53.190932 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.190974 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:46:53.191260 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42967: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42967: connect: Connection refused (error 111)
W20260927 00:46:53.195585 13868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.195586 13870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.195585 13867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:53.196049 13860 server_base.cc:1034] running on GCE node
I20260927 00:46:53.196573 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:53.197638 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:53.198822 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790470013198786 us; error 46 us; skew 500 ppm
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:53.201404 13860 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:46:53.202381 13860 webserver.cc:460] Webserver started at http://127.0.98.60:46779/ using document root <none> and password file <none>
I20260927 00:46:53.202960 13860 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:53.203023 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:53.203193 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:53.205178 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data/instance:
uuid: "641818530243494fb19f791b8a332379"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-l5sb"
server_key: "c720f7fe6c8641c541608f1608920fda"
server_key_iv: "3b3179084170858638a81e519423d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.205741 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal/instance:
uuid: "641818530243494fb19f791b8a332379"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-l5sb"
server_key: "c720f7fe6c8641c541608f1608920fda"
server_key_iv: "3b3179084170858638a81e519423d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.209331 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:53.211684 13877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.212898 13860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:53.213016 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "641818530243494fb19f791b8a332379"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-l5sb"
server_key: "c720f7fe6c8641c541608f1608920fda"
server_key_iv: "3b3179084170858638a81e519423d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.213129 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:53.228995 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:53.231643 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:53.231829 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:53.238481 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42967
I20260927 00:46:53.238507 13929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42967 every 8 connection(s)
I20260927 00:46:53.239349 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:46:53.241937 13930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:53.245443 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:53.247145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 13860
I20260927 00:46:53.247285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469854491772-392-0/minicluster-data/master-2/wal/instance
I20260927 00:46:53.247609 392 external_mini_cluster.cc:1428] Setting key ed0addd446ac6bef6b4aa53c22b825f0
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:46:53.260016 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.247597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.98.62:35339)
0927 00:46:53.247742 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.247747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.248630 (+ 883us) server_negotiation.cc:407] Connection header received
0927 00:46:53.249528 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.249531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.249579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.249680 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.250781 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.251601 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.252905 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.253320 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.253426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.256108 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.256111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.256113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.256137 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.257845 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.258335 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.258343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.258345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.258432 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.258816 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.258818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.258819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.259110 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.259181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.259717 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.259851 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:46:53.260802 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.247996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35339 (local address 127.0.0.1:56256)
0927 00:46:53.248471 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.248521 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.249343 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.249752 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.249758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.250289 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.250632 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.250653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.251708 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.251710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.252774 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.252784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.253459 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.253461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.253593 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.254240 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.254264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.255944 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.257946 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.257954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.257961 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.258236 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.258533 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.258537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.258538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.258717 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.259212 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.259220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.259472 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.259651 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.259968 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20260927 00:46:53.260815 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.248259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.0.98.60:42967)
0927 00:46:53.248984 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.248993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.249023 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:46:53.249357 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.249377 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.249601 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.249811 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.250675 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.251904 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.253001 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.253553 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.253635 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.256549 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.256557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.256576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.256607 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.258420 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.258890 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.258895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.258897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.258973 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.259270 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.259273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.259274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.259500 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.259685 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.259912 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.260045 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":132,"threads_started":1}
I20260927 00:46:53.262154 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:53.271337 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.262644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37717 (local address 127.0.0.1:50242)
0927 00:46:53.262751 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.262769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.262845 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.263383 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.263386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.263571 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.263711 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.263720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.264746 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.264749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.265546 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.265556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.266087 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.266090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.266170 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.266563 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.266602 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.267868 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.269928 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.269931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.269933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.270124 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.270408 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.270411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.270413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.270495 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.270857 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.270860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.270956 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.271060 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.271180 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:46:53.271358 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.262928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.98.61:37717)
0927 00:46:53.263079 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.263083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.263097 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:46:53.263143 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.263149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.263200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.263273 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.263854 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.264642 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.265658 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.265988 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.266087 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.267999 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.268003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.268005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.268047 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.269822 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.270222 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.270227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.270228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.270313 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.270583 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.270588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.270590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.270763 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.270867 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.271119 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.271224 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:46:53.271957 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:46:53.272017 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.263201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.98.62:35339)
0927 00:46:53.263298 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.263302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.263320 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:46:53.263347 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.263349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.263379 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.263439 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.264106 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.264786 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.265842 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.266166 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.266247 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.268606 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.268608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.268610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.268630 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.270119 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.270520 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.270525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.270527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.270604 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.270868 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.270872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.270874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.271029 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.271244 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.271369 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.271881 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:46:53.280984 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.272417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34062)
0927 00:46:53.272564 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.272588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.272651 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.272849 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.272852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.273005 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.273059 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.273064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.274004 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.274007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.274818 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.274826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.275335 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.275337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.275394 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.275829 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.275839 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.277225 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.279203 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.279205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.279207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.279446 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.279800 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.279803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.279805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.280159 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.280625 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.280627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.280701 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.280784 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.280875 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 00:46:53.281172 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.272474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.0.98.60:42967)
0927 00:46:53.272665 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.272670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.272687 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:46:53.272721 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.272724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.272757 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.272842 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.273162 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.273904 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.274951 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.275241 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.275320 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.277349 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.277353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.277355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.277379 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.279087 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.279556 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.279563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.279567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.279699 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.280262 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.280266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.280267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.280508 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.280616 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.280866 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.280973 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:46:53.282418 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 3
I20260927 00:46:53.283664 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.273228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.0.98.61:37717)
0927 00:46:53.273350 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.273354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.273366 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:46:53.273398 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.273400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.273436 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.273511 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.274115 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.274824 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.276162 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.276552 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.276655 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.279641 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.279644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.279646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.279671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.281427 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.281819 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.281825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.281828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.281908 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.282213 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.282217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.282218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.282452 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.282657 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.283399 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.283517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:53.284672 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } attempt: 3
I20260927 00:46:53.284920 13930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: Bootstrap starting.
I20260927 00:46:53.287089 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:46:53.288481 13930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:53.290011 13930 log.cc:824] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:46:53 dist-test-slave-l5sb krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:46:53.291488 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.283002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.0.98.60:42967)
0927 00:46:53.283115 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.283120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.283134 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:46:53.283166 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.283169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.283212 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.283300 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.283761 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.284367 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.285341 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.285639 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.286099 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.287953 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.287957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.287959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.287981 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.289564 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.290125 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.290129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.290130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.290210 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.290568 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.290573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.290576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.290820 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.290922 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.291264 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.291362 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:46:53.291488 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.282782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34074)
0927 00:46:53.283019 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.283034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.283101 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.283316 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.283319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.283517 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.283621 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.283627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.284489 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.284493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.285213 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.285223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.285808 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.285813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.285895 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.286342 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.286356 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.287812 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.289706 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.289710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.289715 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.290009 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.290364 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.290367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.290370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.290465 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.290972 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.290977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.291075 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.291202 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.291333 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 00:46:53.293660 13930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: No bootstrap required, opened a new log
I20260927 00:46:53.294950 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: Bootstrap starting.
I20260927 00:46:53.295027 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.285502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42967 (local address 127.0.0.1:34086)
0927 00:46:53.285680 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.285694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.285761 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.286549 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.286553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.286742 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.286842 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.286857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.287947 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.287951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.288715 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.288726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.289311 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.289314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.289397 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.289830 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.289849 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.291464 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.293474 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.293477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.293479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.293709 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.294023 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.294027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.294030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.294130 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.294553 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.294556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.294642 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.294730 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.294846 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 00:46:53.295611 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.285959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.98.60:42967)
0927 00:46:53.286309 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.286317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.286329 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:46:53.286360 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.286363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.286399 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.286965 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.287089 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.287804 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.288862 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.289192 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.289309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.291639 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.291643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.291645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.291670 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.293361 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.293828 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.293831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.293833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.293924 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.294232 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.294235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.294237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.294428 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.294513 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.295316 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.295433 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260927 00:46:53.297699 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:53.297991 13930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.298233 13930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:53.298291 13930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641818530243494fb19f791b8a332379, State: Initialized, Role: FOLLOWER
I20260927 00:46:53.298600 13854 log.cc:824] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:53.298755 13930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.299098 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: Bootstrap starting.
I20260927 00:46:53.300107 13930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:53.301440 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: No bootstrap required, opened a new log
I20260927 00:46:53.301718 13779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:53.305532 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.291064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.0.98.62:35339)
0927 00:46:53.291168 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.291173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.291848 (+ 675us) server_negotiation.cc:407] Connection header received
0927 00:46:53.292024 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.292030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.292079 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.292154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.293046 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.293823 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.296195 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.296495 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.296584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.301136 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.301140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.301142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.301184 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.302796 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.304111 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.304114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.304116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.304199 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.304509 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.304512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.304514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.304718 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.304817 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.305207 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.305338 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:53.305570 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.306336 13854 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:53.306434 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d1e679b1a14bc09f03294e88d9cc04, State: Initialized, Role: FOLLOWER
W20260927 00:46:53.306799 13957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:46:53.306869 13957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:46:53.306996 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.308384 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:53.304446 13860 master_runner.cc:186] Error getting master registration for 127.0.98.62:35339: OK, instance_id { permanent_uuid: "156873d04adb4056bee5edecb68276bb" instance_seqno: 1790470012820772 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:46:53.308732 13860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:46:53.302806 13779 log.cc:824] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:53.310089 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.290202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.0.98.60:42967)
0927 00:46:53.291637 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.291643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.295118 (+ 3475us) server_negotiation.cc:407] Connection header received
0927 00:46:53.295284 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.295287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.295324 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.295368 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.297075 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.298050 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.300242 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.300547 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.300617 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.306185 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.306188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.306191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.306215 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.307833 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.308300 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.308304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.308305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.308381 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.308781 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.308786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.308791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.309068 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.309165 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.309704 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.309866 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":91,"threads_started":1}
I20260927 00:46:53.311081 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.311204 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.311582 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.311712 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:46:53.313858 13971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:46:53.313987 13971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:46:53.311990 13786 master_runner.cc:186] Error getting master registration for 127.0.98.62:35339: OK, instance_id { permanent_uuid: "156873d04adb4056bee5edecb68276bb" instance_seqno: 1790470012820772 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:46:53.315236 13786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:46:53.318872 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.289701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.0.98.61:37717)
0927 00:46:53.289825 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.289829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.291252 (+ 1423us) server_negotiation.cc:407] Connection header received
0927 00:46:53.291422 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.291425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.291463 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.291515 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.311687 (+ 20172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.312399 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.313471 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.313835 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.313920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.315564 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.315567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.315570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.315596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.317125 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.317620 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.317624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.317626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.317736 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.318030 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.318033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.318035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.318206 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.318351 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.318591 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.318700 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:46:53.323376 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: No bootstrap required, opened a new log
I20260927 00:46:53.326486 13779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.326699 13779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:53.326764 13779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156873d04adb4056bee5edecb68276bb, State: Initialized, Role: FOLLOWER
I20260927 00:46:53.327231 13779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.329330 13779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:46:53.334348 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:46:53.334470 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:46:53.334831 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.335029 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.338388 13711 master_runner.cc:263] No leader master found from master 156873d04adb4056bee5edecb68276bb
I20260927 00:46:53.338469 13711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:46:53.378356 13959 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:53.378547 13959 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.379654 13959 leader_election.cc:288] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 156873d04adb4056bee5edecb68276bb (127.0.98.62:35339), 641818530243494fb19f791b8a332379 (127.0.98.60:42967)
I20260927 00:46:53.380558 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156873d04adb4056bee5edecb68276bb" is_pre_election: true
I20260927 00:46:53.380887 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d1e679b1a14bc09f03294e88d9cc04 in term 0.
I20260927 00:46:53.381696 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641818530243494fb19f791b8a332379" is_pre_election: true
I20260927 00:46:53.382025 13904 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d1e679b1a14bc09f03294e88d9cc04 in term 0.
I20260927 00:46:53.382607 13805 leader_election.cc:302] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156873d04adb4056bee5edecb68276bb, f0d1e679b1a14bc09f03294e88d9cc04; no voters:
I20260927 00:46:53.382920 13959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:53.382992 13959 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:53.383033 13959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:53.384359 13959 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.384711 13959 leader_election.cc:288] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [CANDIDATE]: Term 1 election: Requested vote from peers 156873d04adb4056bee5edecb68276bb (127.0.98.62:35339), 641818530243494fb19f791b8a332379 (127.0.98.60:42967)
I20260927 00:46:53.385680 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156873d04adb4056bee5edecb68276bb"
I20260927 00:46:53.385820 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:53.387096 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d1e679b1a14bc09f03294e88d9cc04 in term 1.
I20260927 00:46:53.387471 13805 leader_election.cc:302] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156873d04adb4056bee5edecb68276bb, f0d1e679b1a14bc09f03294e88d9cc04; no voters:
I20260927 00:46:53.387807 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641818530243494fb19f791b8a332379"
I20260927 00:46:53.387931 13904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:53.389199 13904 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d1e679b1a14bc09f03294e88d9cc04 in term 1.
I20260927 00:46:53.389724 13959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:53.390008 13959 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [term 1 LEADER]: Becoming Leader. State: Replica: f0d1e679b1a14bc09f03294e88d9cc04, State: Running, Role: LEADER
I20260927 00:46:53.390363 13959 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } }
I20260927 00:46:53.392127 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d1e679b1a14bc09f03294e88d9cc04. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.392225 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:53.395061 13988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:53.396050 13988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:53.402240 13904 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [term 1 FOLLOWER]: Refusing update from remote peer f0d1e679b1a14bc09f03294e88d9cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:53.403057 13986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:53.407881 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d1e679b1a14bc09f03294e88d9cc04. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.408002 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.411012 13753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [term 1 FOLLOWER]: Refusing update from remote peer f0d1e679b1a14bc09f03294e88d9cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:53.411612 13986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:53.415491 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d1e679b1a14bc09f03294e88d9cc04. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.415606 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.420240 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.420363 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:53.421434 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.421553 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.421974 13989 mvcc.cc:205] Tried to move back new op lower bound from 7333765174887325696 to 7333765174849843200. Current Snapshot: MvccSnapshot[applied={T|T < 7333765174887325696}]
I20260927 00:46:53.425572 13992 mvcc.cc:205] Tried to move back new op lower bound from 7333765174887325696 to 7333765174849843200. Current Snapshot: MvccSnapshot[applied={T|T < 7333765174887325696}]
I20260927 00:46:53.426604 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.426709 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:53.427678 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "156873d04adb4056bee5edecb68276bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } peers { permanent_uuid: "f0d1e679b1a14bc09f03294e88d9cc04" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37717 } } peers { permanent_uuid: "641818530243494fb19f791b8a332379" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42967 } } }
I20260927 00:46:53.427788 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:46:53.430738 13990 mvcc.cc:205] Tried to move back new op lower bound from 7333765174887325696 to 7333765174849843200. Current Snapshot: MvccSnapshot[applied={T|T < 7333765174887325696}]
I20260927 00:46:53.434055 13988 catalog_manager.cc:1347] Generated new cluster ID: 49fadba1bb4141da9ccc46ae6f95d37e
I20260927 00:46:53.434132 13988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:53.466193 13988 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:53.467593 13988 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:53.485272 13988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d1e679b1a14bc09f03294e88d9cc04: Generated new TSK 0
I20260927 00:46:53.486382 13988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:46:54.308245 13957 catalog_manager.cc:1259] Loaded cluster ID: 49fadba1bb4141da9ccc46ae6f95d37e
I20260927 00:46:54.308337 13957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: loading cluster ID for follower catalog manager: success
I20260927 00:46:54.310542 13957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: acquiring CA information for follower catalog manager: success
I20260927 00:46:54.311777 13957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 641818530243494fb19f791b8a332379: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:46:54.335765 13984 catalog_manager.cc:1259] Loaded cluster ID: 49fadba1bb4141da9ccc46ae6f95d37e
I20260927 00:46:54.335892 13984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: loading cluster ID for follower catalog manager: success
I20260927 00:46:54.337541 13984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: acquiring CA information for follower catalog manager: success
I20260927 00:46:54.338706 13984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 156873d04adb4056bee5edecb68276bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:46:54.563710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13711
I20260927 00:46:54.569971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13786
I20260927 00:46:54.575734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 13860
2026-09-27T00:46:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3104 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:54 dist-test-slave-l5sb krb5kdc[14002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:54 dist-test-slave-l5sb krb5kdc[14002](info): set up 2 sockets
Sep 27 00:46:54 dist-test-slave-l5sb krb5kdc[14002](info): commencing operation
krb5kdc: starting...
W20260927 00:46:55.610092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:55 dist-test-slave-l5sb krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:55.729293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:55.836807 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:55.837117 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:55.837193 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:55.841177 14015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:55.841260 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:55.841297 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:55.841336 14015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:55.841370 14015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:55.846570 14015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:55.847564 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:55.848528 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:55.856711 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:55.856698 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:55.856700 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:55.856844 14015 server_base.cc:1034] running on GCE node
I20260927 00:46:55.857632 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:55.858741 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:55.859898 14015 hybrid_clock.cc:648] HybridClock initialized: now 1790470015859889 us; error 36 us; skew 500 ppm
Sep 27 00:46:55 dist-test-slave-l5sb krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:55.862622 14015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:55.863480 14015 webserver.cc:460] Webserver started at http://127.0.98.62:43999/ using document root <none> and password file <none>
I20260927 00:46:55.864045 14015 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:55.864137 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:55.864416 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:55.866520 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "f21fc2262d3f4bffa72933ee625ded2b"
format_stamp: "Formatted at 2026-09-27 00:46:55 on dist-test-slave-l5sb"
server_key: "2561381cc5b1a69406d1dbd985ea0937"
server_key_iv: "2565939b7dd3704bdf492d284a7ae80e"
server_key_version: "encryptionkey@0"
I20260927 00:46:55.867097 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "f21fc2262d3f4bffa72933ee625ded2b"
format_stamp: "Formatted at 2026-09-27 00:46:55 on dist-test-slave-l5sb"
server_key: "2561381cc5b1a69406d1dbd985ea0937"
server_key_iv: "2565939b7dd3704bdf492d284a7ae80e"
server_key_version: "encryptionkey@0"
I20260927 00:46:55.870734 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:46:55.873054 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:55.874306 14015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:55.874449 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "f21fc2262d3f4bffa72933ee625ded2b"
format_stamp: "Formatted at 2026-09-27 00:46:55 on dist-test-slave-l5sb"
server_key: "2561381cc5b1a69406d1dbd985ea0937"
server_key_iv: "2565939b7dd3704bdf492d284a7ae80e"
server_key_version: "encryptionkey@0"
I20260927 00:46:55.874558 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:55.887905 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:55.890774 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:55.890969 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:55.897810 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34211
I20260927 00:46:55.897872 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34211 every 8 connection(s)
I20260927 00:46:55.898789 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:55.901494 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:55.905154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14015
I20260927 00:46:55.905292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:55.905553 392 external_mini_cluster.cc:1428] Setting key 0f4b1236ef9b8cbe2cfbf1f3afc0231d
I20260927 00:46:55.907183 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b: Bootstrap starting.
I20260927 00:46:55.909602 14083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:55.910370 14083 log.cc:824] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:55.912402 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b: No bootstrap required, opened a new log
Sep 27 00:46:55 dist-test-slave-l5sb krb5kdc[14002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:55.916626 14083 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } }
I20260927 00:46:55.916833 14083 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:55.916932 14083 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21fc2262d3f4bffa72933ee625ded2b, State: Initialized, Role: FOLLOWER
I20260927 00:46:55.917415 14083 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } }
I20260927 00:46:55.917538 14083 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:55.917630 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:55.917748 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:55.918808 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } }
I20260927 00:46:55.919214 14083 leader_election.cc:302] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21fc2262d3f4bffa72933ee625ded2b; no voters:
I20260927 00:46:55.919559 14083 leader_election.cc:288] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:55.919715 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:55.919919 14088 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [term 1 LEADER]: Becoming Leader. State: Replica: f21fc2262d3f4bffa72933ee625ded2b, State: Running, Role: LEADER
I20260927 00:46:55.920229 14088 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } }
I20260927 00:46:55.920302 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:55.906825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.98.62:34211)
0927 00:46:55.907216 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:55.907231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:55.907266 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:46:55.907924 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:55.907941 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:55.908256 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:55.908522 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:55.909397 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:55.910462 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:55.911578 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:55.912033 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:55.912148 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:55.914906 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:55.914911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:55.914925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:55.914950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:55.916677 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:55.917202 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:55.917207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:55.917211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:55.917320 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:55.917656 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:55.917658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:55.917659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:55.918161 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:55.918379 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:55.918753 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 00:46:55.919211 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260927 00:46:55.920855 14083 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:55.921409 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21fc2262d3f4bffa72933ee625ded2b. Latest consensus state: current_term: 1 leader_uuid: "f21fc2262d3f4bffa72933ee625ded2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } } }
I20260927 00:46:55.921514 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:55.921862 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21fc2262d3f4bffa72933ee625ded2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21fc2262d3f4bffa72933ee625ded2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34211 } } }
I20260927 00:46:55.921965 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:55.922466 14092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:55.923354 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:46:55.926276 14092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:55.931869 14092 catalog_manager.cc:1347] Generated new cluster ID: d705e744c4cb4f84917a640bace10107
I20260927 00:46:55.931939 14092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:55.959952 14092 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:55.960987 14092 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:55.968204 14092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21fc2262d3f4bffa72933ee625ded2b: Generated new TSK 0
I20260927 00:46:55.968899 14092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:46:55.979888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14015
2026-09-27T00:46:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1401 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:56 dist-test-slave-l5sb krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:56 dist-test-slave-l5sb krb5kdc[14110](info): set up 2 sockets
Sep 27 00:46:56 dist-test-slave-l5sb krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260927 00:46:57.050015 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:57.168902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:57.278986 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:57.279245 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:57.279284 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:57.283388 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:57.283443 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:57.283488 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:57.283528 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:57.283568 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:57.288755 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:57.289825 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:57.290844 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:57.300735 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.300727 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.300727 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.301167 14123 server_base.cc:1034] running on GCE node
I20260927 00:46:57.301645 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:57.302726 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:57.303908 14123 hybrid_clock.cc:648] HybridClock initialized: now 1790470017303877 us; error 41 us; skew 500 ppm
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:57.306707 14123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:57.307654 14123 webserver.cc:460] Webserver started at http://127.0.98.62:41829/ using document root <none> and password file <none>
I20260927 00:46:57.308210 14123 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:57.308303 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:57.308550 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:57.310652 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "edcf9d6c1c5541b6b965b1f270fc55fd"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "4d95516476c786c32c91ce1180a02b1b"
server_key_iv: "1866bc6396899d2d2af3364e4ca6f677"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.311218 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "edcf9d6c1c5541b6b965b1f270fc55fd"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "4d95516476c786c32c91ce1180a02b1b"
server_key_iv: "1866bc6396899d2d2af3364e4ca6f677"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.314940 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:46:57.317414 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.318773 14123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:57.318955 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "edcf9d6c1c5541b6b965b1f270fc55fd"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "4d95516476c786c32c91ce1180a02b1b"
server_key_iv: "1866bc6396899d2d2af3364e4ca6f677"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.319069 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:57.332976 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:57.335773 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:57.335971 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:57.343159 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46323
I20260927 00:46:57.343437 14190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46323 every 8 connection(s)
I20260927 00:46:57.344147 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:57.344719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14123
I20260927 00:46:57.344867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:57.345144 392 external_mini_cluster.cc:1428] Setting key 67bf7b4e5cedace906bbe43baa8a0131
I20260927 00:46:57.347294 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.352706 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap starting.
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:57.354981 14191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.355880 14191 log.cc:824] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:57.358433 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: No bootstrap required, opened a new log
I20260927 00:46:57.359275 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.346560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.0.98.62:46323)
0927 00:46:57.347001 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.347015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.347051 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:46:57.347757 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.347774 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.348098 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.348436 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.349274 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.350328 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.351175 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.351641 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.351807 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.354375 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.354380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.354392 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.354418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.356460 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.356993 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.356997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.357002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.357126 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.357422 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.357424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.357425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.357853 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.358001 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.358275 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.358548 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260927 00:46:57.362352 14191 raft_consensus.cc:348] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:57.362593 14191 raft_consensus.cc:374] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.362671 14191 raft_consensus.cc:729] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.363149 14191 consensus_queue.cc:260] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:57.363325 14191 raft_consensus.cc:388] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:57.363399 14191 raft_consensus.cc:482] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:57.363503 14191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.364365 14191 raft_consensus.cc:504] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:57.364701 14191 leader_election.cc:302] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edcf9d6c1c5541b6b965b1f270fc55fd; no voters:
I20260927 00:46:57.365026 14191 leader_election.cc:288] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:57.365137 14196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:57.365381 14196 raft_consensus.cc:686] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 LEADER]: Becoming Leader. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Running, Role: LEADER
I20260927 00:46:57.365751 14196 consensus_queue.cc:237] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:57.366367 14191 sys_catalog.cc:564] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:57.367421 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader edcf9d6c1c5541b6b965b1f270fc55fd. Latest consensus state: current_term: 1 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:57.367552 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:57.367843 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:57.367950 14205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:57.368021 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:57.370615 14205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:57.375916 14205 catalog_manager.cc:1347] Generated new cluster ID: 04d809aeb5a24b0c916906c13b26c431
I20260927 00:46:57.375981 14205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:57.388958 14205 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:57.390316 14205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:57.407186 14205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Generated new TSK 0
I20260927 00:46:57.407989 14205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:57.444511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:57.567786 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:57.568032 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:57.568073 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:57.572093 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:57.572145 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:57.572201 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:57.577155 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:57.578150 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:57.579171 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:57.588670 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.588608 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.588666 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.589032 14217 server_base.cc:1034] running on GCE node
I20260927 00:46:57.589444 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:57.590184 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:57.591355 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790470017591321 us; error 46 us; skew 500 ppm
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:57.594059 14217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:57.594907 14217 webserver.cc:460] Webserver started at http://127.0.98.1:46695/ using document root <none> and password file <none>
I20260927 00:46:57.595500 14217 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:57.595593 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:57.595817 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:57.597879 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e9f403be31740f5bb5f324a5cc118a7"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "cc48be537dcd3a97bc3a554a05563893"
server_key_iv: "2cfd88c49cf08f7f8d6bbb028142dd45"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.598449 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e9f403be31740f5bb5f324a5cc118a7"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "cc48be537dcd3a97bc3a554a05563893"
server_key_iv: "2cfd88c49cf08f7f8d6bbb028142dd45"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.602253 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 00:46:57.604612 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.605811 14217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:57.605945 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "7e9f403be31740f5bb5f324a5cc118a7"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "cc48be537dcd3a97bc3a554a05563893"
server_key_iv: "2cfd88c49cf08f7f8d6bbb028142dd45"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.606060 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:57.627964 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:57.630903 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:57.631106 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:57.631701 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:57.632542 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:57.632611 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.632682 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:57.632735 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.641829 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38179
I20260927 00:46:57.641880 14345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38179 every 8 connection(s)
I20260927 00:46:57.642747 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:57.651228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14217
I20260927 00:46:57.651372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:57.651674 392 external_mini_cluster.cc:1428] Setting key e662947957e710bd96107f602f7c12b9
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:57.660470 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.644563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47429 (local address 127.0.98.62:46323)
0927 00:46:57.644700 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.644704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.645473 (+ 769us) server_negotiation.cc:407] Connection header received
0927 00:46:57.646862 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.646865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.646928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.647031 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.648196 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.648741 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.649930 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.650174 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.650286 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.656969 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.656972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.656974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.656997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.658445 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.658959 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.658967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.658968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.659040 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.659396 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.659399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.659401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.659580 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.659690 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.660177 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.660303 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:46:57.661280 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.644881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:47429)
0927 00:46:57.645320 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.645362 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.646642 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.647210 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.647217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.647569 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.648036 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.648050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.648872 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.648880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.649754 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.649812 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.650326 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.650330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.650498 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.655037 (+ 4539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.655063 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.656779 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.658573 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.658579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.658595 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.658849 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.659140 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.659144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.659146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.659294 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.659687 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.659692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.659941 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.660121 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.660451 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260927 00:46:57.662513 14346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:57.662817 14346 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:57.663429 14346 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:57.665158 14155 ts_manager.cc:194] Registered new tserver with Master: 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:57.666716 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:57.679123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:57.788380 14351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:57.788652 14351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:57.788727 14351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:57.792654 14351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:57.792742 14351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:57.792829 14351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:57.797912 14351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:57.798990 14351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:57.800040 14351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:57.809366 14359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.809427 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.809388 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.809724 14351 server_base.cc:1034] running on GCE node
I20260927 00:46:57.810187 14351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:57.810838 14351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:57.812036 14351 hybrid_clock.cc:648] HybridClock initialized: now 1790470017812023 us; error 24 us; skew 500 ppm
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:57.814893 14351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:57.815738 14351 webserver.cc:460] Webserver started at http://127.0.98.2:43415/ using document root <none> and password file <none>
I20260927 00:46:57.816282 14351 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:57.816340 14351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:57.816568 14351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:57.818817 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "48c640317f17476aae0cc25a1c5a36b4"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "c8bc77b1bd7169a3567713971c42a033"
server_key_iv: "ac2029f67420080cd795803c7b2c6691"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.819366 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48c640317f17476aae0cc25a1c5a36b4"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "c8bc77b1bd7169a3567713971c42a033"
server_key_iv: "ac2029f67420080cd795803c7b2c6691"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.823208 14351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:46:57.825546 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.826769 14351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:57.826891 14351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "48c640317f17476aae0cc25a1c5a36b4"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "c8bc77b1bd7169a3567713971c42a033"
server_key_iv: "ac2029f67420080cd795803c7b2c6691"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.826993 14351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:57.852166 14351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:57.854982 14351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:57.855176 14351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:57.855762 14351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:57.856588 14351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:57.856634 14351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.856705 14351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:57.856745 14351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.865437 14351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38525
I20260927 00:46:57.865460 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38525 every 8 connection(s)
I20260927 00:46:57.866413 14351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:57.875272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14351
I20260927 00:46:57.875406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:57.875667 392 external_mini_cluster.cc:1428] Setting key e2965d9b975b43897c5d39bd36688a19
Sep 27 00:46:57 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:57.880195 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.868319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51707 (local address 127.0.98.62:46323)
0927 00:46:57.868444 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.868448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.869255 (+ 807us) server_negotiation.cc:407] Connection header received
0927 00:46:57.870639 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.870643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.870698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.870783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.871929 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.872460 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.873584 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.873887 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.873994 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.876503 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.876506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.876509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.876532 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.878106 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.878692 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.878694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.878695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.878789 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.879150 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.879153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.879155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.879329 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.879430 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.879910 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.880021 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:46:57.880944 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.868697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:51707)
0927 00:46:57.869115 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.869157 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.870416 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.870966 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.870973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.871315 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.871780 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.871793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.872598 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.872606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.873456 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.873464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.874033 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.874037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.874212 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.874717 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.874732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.876336 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.878212 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.878221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.878237 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.878598 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.878890 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.878894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.878896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.879038 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.879441 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.879449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.879681 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.879868 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.880172 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260927 00:46:57.882056 14480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:57.882311 14480 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:57.882906 14480 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:57.884024 14155 ts_manager.cc:194] Registered new tserver with Master: 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
I20260927 00:46:57.884598 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51707
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:57.905401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:58.013253 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:58.013528 14485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:58.013633 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:58.017694 14485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:58.017788 14485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:58.017889 14485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:58.023283 14485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:58.024292 14485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:58.025228 14485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:58.034365 14490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.034511 14493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.034554 14485 server_base.cc:1034] running on GCE node
W20260927 00:46:58.034369 14491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.035156 14485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:58.035852 14485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:58.037037 14485 hybrid_clock.cc:648] HybridClock initialized: now 1790470018037003 us; error 43 us; skew 500 ppm
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:58.039753 14485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:58.040637 14485 webserver.cc:460] Webserver started at http://127.0.98.3:35061/ using document root <none> and password file <none>
I20260927 00:46:58.041186 14485 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:58.041245 14485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:58.041477 14485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:58.043648 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "af64511fa3a242b5947eb4ddc59465f8"
format_stamp: "Formatted at 2026-09-27 00:46:58 on dist-test-slave-l5sb"
server_key: "92b7ee1ad0e4202d14616cbe07f15f53"
server_key_iv: "3645ee616c8b920237588d0bb5a0f171"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.044217 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af64511fa3a242b5947eb4ddc59465f8"
format_stamp: "Formatted at 2026-09-27 00:46:58 on dist-test-slave-l5sb"
server_key: "92b7ee1ad0e4202d14616cbe07f15f53"
server_key_iv: "3645ee616c8b920237588d0bb5a0f171"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.047930 14485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:46:58.050282 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.051498 14485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:58.051628 14485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "af64511fa3a242b5947eb4ddc59465f8"
format_stamp: "Formatted at 2026-09-27 00:46:58 on dist-test-slave-l5sb"
server_key: "92b7ee1ad0e4202d14616cbe07f15f53"
server_key_iv: "3645ee616c8b920237588d0bb5a0f171"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.051735 14485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:58.063294 14485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:58.066004 14485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:58.066207 14485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:58.066756 14485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:58.067646 14485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:58.067713 14485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.067781 14485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:58.067830 14485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.076733 14485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44553
I20260927 00:46:58.076757 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44553 every 8 connection(s)
I20260927 00:46:58.077643 14485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:58.081132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14485
I20260927 00:46:58.081223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:58.081485 392 external_mini_cluster.cc:1428] Setting key b89dc430face0a073e4b46942ddb7579
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:58.091377 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.079429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34805 (local address 127.0.98.62:46323)
0927 00:46:58.079577 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.079582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.080441 (+ 859us) server_negotiation.cc:407] Connection header received
0927 00:46:58.081826 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.081834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.081885 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.081959 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.083496 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.084146 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.085248 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.085449 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.085524 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.087847 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.087850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.087852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.087879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.089249 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.089866 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.089870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.089872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.089969 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.090331 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.090333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.090335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.090547 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.090632 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.091119 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.091238 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:46:58.092182 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.079872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:34805)
0927 00:46:58.080289 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.080331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.081572 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.082139 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.082146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.082662 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:58.083339 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.083358 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.084259 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.084261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.085143 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.085152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.085566 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.085568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.085702 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.086282 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:58.086298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:58.087698 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:58.089400 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.089405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.089417 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.089687 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.090090 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.090093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.090094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.090234 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.090643 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:58.090648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:58.090887 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.091052 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.091314 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":119,"threads_started":1}
I20260927 00:46:58.093393 14614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:58.093647 14614 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:58.094264 14614 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:58.095368 14154 ts_manager.cc:194] Registered new tserver with Master: af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553)
I20260927 00:46:58.095953 14154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34805
I20260927 00:46:58.108122 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:58.117341 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.110220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.0.98.62:46323)
0927 00:46:58.110363 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.110368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.110611 (+ 243us) server_negotiation.cc:407] Connection header received
0927 00:46:58.110736 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.110738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.110777 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.110835 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.111510 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.112055 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.112859 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.113110 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.113193 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.114452 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.114455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.114457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.114482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.115956 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.116314 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.116316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.116317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.116388 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.116601 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.116603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.116604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.116799 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.116911 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.117088 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.117175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:46:58.120092 14154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:58.123106 14154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:58.137056 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.131723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.0.98.2:38525)
0927 00:46:58.132262 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.132275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.132297 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:46:58.132370 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.132375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.132581 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.133113 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.133281 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134451 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.135797 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.136343 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.136471 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.136630 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.136802 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.136908 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":119,"threads_started":1}
I20260927 00:46:58.137053 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.130984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38525 (local address 127.0.0.1:41622)
0927 00:46:58.131529 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.131574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.131697 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.132759 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.132765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.132807 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.132967 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.132973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134602 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.134605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.135689 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.135696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.136468 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.136472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.136590 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.136626 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.136769 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.136901 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"spinlock_wait_cycles":11392,"thread_start_us":149,"threads_started":1}
I20260927 00:46:58.137312 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.131173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44553 (local address 127.0.0.1:59220)
0927 00:46:58.131915 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.131928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.131999 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.132758 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.132764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.132800 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.132938 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.132944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134160 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.134163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.135463 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.135474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.136725 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.136729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.136818 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.136843 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.136941 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.137026 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":53,"spinlock_wait_cycles":2944,"thread_start_us":89,"threads_started":1}
I20260927 00:46:58.137867 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.131364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.0.98.3:44553)
0927 00:46:58.131999 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.132012 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.132037 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:58.132115 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.132120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.132384 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.132558 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.133065 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134033 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.135600 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.136421 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.136531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.136659 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.137118 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.137690 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":95,"threads_started":1}
I20260927 00:46:58.138417 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.132482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.98.1:38179)
0927 00:46:58.132787 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.132803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.132823 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:46:58.132893 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.132898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.133092 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.133226 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.133429 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134536 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.136106 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.136901 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.137992 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.138162 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.138214 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.138282 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260927 00:46:58.139334 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.131747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.0.1:46570)
0927 00:46:58.132119 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.132134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.132206 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.133228 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.133230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.133249 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.133334 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.133341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.134842 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.134845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.135955 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.135965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.137035 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.137038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.137132 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.137152 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.137202 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.137258 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":26,"thread_start_us":85,"threads_started":1}
I20260927 00:46:58.140580 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (DEFAULT_TABLE table=test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:58.140859 14548 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (DEFAULT_TABLE table=test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:58.141611 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea22c1cb4e84c6c8ca6ad0015023453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:58.141942 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea22c1cb4e84c6c8ca6ad0015023453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:58.141929 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (DEFAULT_TABLE table=test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:58.142875 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea22c1cb4e84c6c8ca6ad0015023453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:58.147969 14631 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Bootstrap starting.
I20260927 00:46:58.149317 14632 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Bootstrap starting.
I20260927 00:46:58.150048 14633 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap starting.
I20260927 00:46:58.151000 14631 tablet_bootstrap.cc:654] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:58.151412 14632 tablet_bootstrap.cc:654] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:58.151666 14633 tablet_bootstrap.cc:654] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:58.151942 14631 log.cc:824] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:58.152585 14632 log.cc:824] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:58.152806 14633 log.cc:824] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:58.154604 14632 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: No bootstrap required, opened a new log
I20260927 00:46:58.154659 14633 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: No bootstrap required, opened a new log
I20260927 00:46:58.154659 14631 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: No bootstrap required, opened a new log
I20260927 00:46:58.154824 14632 ts_tablet_manager.cc:1397] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:46:58.154826 14633 ts_tablet_manager.cc:1397] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:46:58.154826 14631 ts_tablet_manager.cc:1397] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:46:58.157948 14632 raft_consensus.cc:348] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.158213 14632 raft_consensus.cc:374] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:58.158294 14632 raft_consensus.cc:729] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af64511fa3a242b5947eb4ddc59465f8, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.158776 14632 consensus_queue.cc:260] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.158764 14633 raft_consensus.cc:348] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.158764 14631 raft_consensus.cc:348] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.159025 14631 raft_consensus.cc:374] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:58.159061 14633 raft_consensus.cc:374] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:58.159091 14631 raft_consensus.cc:729] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c640317f17476aae0cc25a1c5a36b4, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.159143 14633 raft_consensus.cc:729] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9f403be31740f5bb5f324a5cc118a7, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.159544 14631 consensus_queue.cc:260] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.159580 14633 consensus_queue.cc:260] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.160030 14632 ts_tablet_manager.cc:1428] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:46:58.160487 14631 ts_tablet_manager.cc:1428] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:46:58.160487 14633 ts_tablet_manager.cc:1428] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:46:58.160575 14614 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
I20260927 00:46:58.160575 14346 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
I20260927 00:46:58.160835 14480 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
I20260927 00:46:58.228156 14637 raft_consensus.cc:482] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:58.228391 14637 raft_consensus.cc:504] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.229547 14637 leader_election.cc:288] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525), 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:58.235097 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.229965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50403 (local address 127.0.98.2:38525)
0927 00:46:58.230109 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.230114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.230153 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:46:58.230230 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.230233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.230287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.230371 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.230965 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.231808 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.233726 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.234368 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.234461 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.234523 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.234893 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.234965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:46:58.235097 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.230245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39665 (local address 127.0.98.1:38179)
0927 00:46:58.230370 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.230375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.230621 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:46:58.230724 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.230727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.230777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.230854 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.231120 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.231826 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.233726 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.234340 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.234458 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.234540 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.234893 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.234965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:46:58.235138 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.229839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38525 (local address 127.0.98.3:50403)
0927 00:46:58.229961 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.229978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.230076 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.230451 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.230454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.230475 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.230627 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.230634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.232032 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.232035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.233596 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.233607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.234475 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.234478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.234716 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.234787 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.234864 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.234977 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":338}
I20260927 00:46:58.235146 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.230152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.98.3:39665)
0927 00:46:58.230526 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.230539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.230643 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.230897 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.230899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.230913 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.230993 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.230997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.232032 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.232035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.233604 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.233609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.234475 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.234478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.234693 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.234783 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.234864 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.234974 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":237,"thread_start_us":141,"threads_started":1}
I20260927 00:46:58.235787 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea22c1cb4e84c6c8ca6ad0015023453" candidate_uuid: "af64511fa3a242b5947eb4ddc59465f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c640317f17476aae0cc25a1c5a36b4" is_pre_election: true
I20260927 00:46:58.235795 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea22c1cb4e84c6c8ca6ad0015023453" candidate_uuid: "af64511fa3a242b5947eb4ddc59465f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9f403be31740f5bb5f324a5cc118a7" is_pre_election: true
I20260927 00:46:58.236090 14300 raft_consensus.cc:2454] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af64511fa3a242b5947eb4ddc59465f8 in term 0.
I20260927 00:46:58.236094 14434 raft_consensus.cc:2454] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af64511fa3a242b5947eb4ddc59465f8 in term 0.
I20260927 00:46:58.236531 14501 leader_election.cc:302] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8; no voters:
I20260927 00:46:58.236804 14637 raft_consensus.cc:2781] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:58.236887 14637 raft_consensus.cc:482] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:58.236941 14637 raft_consensus.cc:3037] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.238082 14637 raft_consensus.cc:504] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.238407 14637 leader_election.cc:288] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [CANDIDATE]: Term 1 election: Requested vote from peers 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525), 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:58.238790 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea22c1cb4e84c6c8ca6ad0015023453" candidate_uuid: "af64511fa3a242b5947eb4ddc59465f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c640317f17476aae0cc25a1c5a36b4"
I20260927 00:46:58.238922 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea22c1cb4e84c6c8ca6ad0015023453" candidate_uuid: "af64511fa3a242b5947eb4ddc59465f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9f403be31740f5bb5f324a5cc118a7"
I20260927 00:46:58.238952 14434 raft_consensus.cc:3037] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.239086 14300 raft_consensus.cc:3037] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.240033 14434 raft_consensus.cc:2454] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af64511fa3a242b5947eb4ddc59465f8 in term 1.
I20260927 00:46:58.240088 14300 raft_consensus.cc:2454] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af64511fa3a242b5947eb4ddc59465f8 in term 1.
I20260927 00:46:58.240332 14501 leader_election.cc:302] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8; no voters:
I20260927 00:46:58.240507 14637 raft_consensus.cc:2781] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:58.240715 14637 raft_consensus.cc:686] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 1 LEADER]: Becoming Leader. State: Replica: af64511fa3a242b5947eb4ddc59465f8, State: Running, Role: LEADER
I20260927 00:46:58.241047 14637 consensus_queue.cc:237] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.244282 14153 catalog_manager.cc:5495] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af64511fa3a242b5947eb4ddc59465f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } health_report { overall_health: UNKNOWN } } }
I20260927 00:46:58.257948 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.254297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.0.98.3:44553)
0927 00:46:58.254433 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.254437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.254453 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:46:58.254670 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.254673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.254727 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.254838 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:58.255131 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.255625 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.256465 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.256683 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.256790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.257106 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.257510 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.257650 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.257697 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.257749 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:46:58.263705 14300 raft_consensus.cc:1261] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Refusing update from remote peer af64511fa3a242b5947eb4ddc59465f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:58.263713 14434 raft_consensus.cc:1261] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Refusing update from remote peer af64511fa3a242b5947eb4ddc59465f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:58.264411 14637 consensus_queue.cc:1035] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:58.264653 14641 consensus_queue.cc:1035] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:58.272097 14645 mvcc.cc:205] Tried to move back new op lower bound from 7333765194801119232 to 7333765194717937664. Current Snapshot: MvccSnapshot[applied={T|T < 7333765194801119232}]
I20260927 00:46:58.279242 14155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:58.279412 14155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:58.281548 14155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Sending DeleteTablet for 3 replicas of tablet 7ea22c1cb4e84c6c8ca6ad0015023453
I20260927 00:46:58.282328 14548 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59220
I20260927 00:46:58.282356 14280 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46570
I20260927 00:46:58.282501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14217
I20260927 00:46:58.282790 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41622
I20260927 00:46:58.283856 14653 tablet_replica.cc:330] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: stopping tablet replica
I20260927 00:46:58.284130 14653 raft_consensus.cc:2229] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:58.284147 14654 tablet_replica.cc:330] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: stopping tablet replica
I20260927 00:46:58.284428 14654 raft_consensus.cc:2229] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:58.284458 14653 raft_consensus.cc:2258] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:58.284739 14654 raft_consensus.cc:2258] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:58.285760 14653 ts_tablet_manager.cc:1905] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:58.287974 14653 ts_tablet_manager.cc:1918] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:58.288089 14653 log.cc:1196] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/7ea22c1cb4e84c6c8ca6ad0015023453
I20260927 00:46:58.288380 14653 ts_tablet_manager.cc:1939] T 7ea22c1cb4e84c6c8ca6ad0015023453 P af64511fa3a242b5947eb4ddc59465f8: Deleting consensus metadata
I20260927 00:46:58.289430 14141 catalog_manager.cc:4841] TS af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (table test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]) successfully deleted
W20260927 00:46:58.290184 14139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38179 (error 108)
I20260927 00:46:58.290401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14351
I20260927 00:46:58.290746 14654 ts_tablet_manager.cc:1905] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:58.291502 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.291243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.0.1:46586)
0927 00:46:58.291367 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.291435 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:46:58.291680 14139 catalog_manager.cc:4584] TS 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ea22c1cb4e84c6c8ca6ad0015023453: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38179: connect: Connection refused (error 111)
I20260927 00:46:58.299110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14485
I20260927 00:46:58.300383 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.300151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38525 (local address 127.0.0.1:41628)
0927 00:46:58.300257 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.300308 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 00:46:58.304942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14123
I20260927 00:46:58.311344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--webserver_interface=127.0.98.62
--webserver_port=41829
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:58.419747 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:58.420042 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:58.420126 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:58.424757 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:58.424842 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:58.424881 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:58.424927 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:58.424962 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:58.430200 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:58.431252 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:58.432253 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:58.445294 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.445312 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.445294 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.445354 14656 server_base.cc:1034] running on GCE node
I20260927 00:46:58.446185 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:58.447429 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:58.448621 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790470018448607 us; error 32 us; skew 500 ppm
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:58.451501 14656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:58.452353 14656 webserver.cc:460] Webserver started at http://127.0.98.62:41829/ using document root <none> and password file <none>
I20260927 00:46:58.452940 14656 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:58.453053 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:58.456974 14656 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:58.459090 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.460253 14656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:46:58.460395 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "edcf9d6c1c5541b6b965b1f270fc55fd"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "4d95516476c786c32c91ce1180a02b1b"
server_key_iv: "1866bc6396899d2d2af3364e4ca6f677"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.460987 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:58.478550 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:58.481477 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:58.481678 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:58.488843 14656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46323
I20260927 00:46:58.489125 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46323 every 8 connection(s)
I20260927 00:46:58.490000 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:58.494302 14725 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:46:58.495913 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap starting.
I20260927 00:46:58.497439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14656
I20260927 00:46:58.497908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38179
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=46695
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:58.518483 14725 log.cc:824] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:58.529183 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:58.529627 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap complete.
I20260927 00:46:58.535096 14725 raft_consensus.cc:348] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:58.535604 14725 raft_consensus.cc:729] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.536139 14725 consensus_queue.cc:260] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:58.536278 14725 raft_consensus.cc:388] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:58.536350 14725 raft_consensus.cc:482] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:58.536439 14725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:46:58.538872 14725 raft_consensus.cc:504] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:58.539235 14725 leader_election.cc:302] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edcf9d6c1c5541b6b965b1f270fc55fd; no voters:
I20260927 00:46:58.539572 14725 leader_election.cc:288] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:46:58.539799 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:46:58.540212 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 LEADER]: Becoming Leader. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Running, Role: LEADER
I20260927 00:46:58.540556 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:58.540891 14725 sys_catalog.cc:564] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:58.541953 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:58.542085 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:58.542706 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader edcf9d6c1c5541b6b965b1f270fc55fd. Latest consensus state: current_term: 2 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:58.542795 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:58.542827 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:58.546972 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:58.547482 14738 catalog_manager.cc:1259] Loaded cluster ID: 04d809aeb5a24b0c916906c13b26c431
I20260927 00:46:58.547552 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:58.548668 14738 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:46:58.551429 14738 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:46:58.555828 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:58.556490 14738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Loaded TSK: 0
I20260927 00:46:58.557399 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:46:58.629966 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:58.630206 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:58.630272 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:58.634253 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:58.634305 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:58.634392 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:58.639567 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46695
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:58.640633 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:58.641755 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:58.651412 14754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.651475 14753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.651412 14756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.652199 14727 server_base.cc:1034] running on GCE node
I20260927 00:46:58.652617 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:58.653261 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:58.654433 14727 hybrid_clock.cc:648] HybridClock initialized: now 1790470018654403 us; error 51 us; skew 500 ppm
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:58.657301 14727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:58.658207 14727 webserver.cc:460] Webserver started at http://127.0.98.1:46695/ using document root <none> and password file <none>
I20260927 00:46:58.658768 14727 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:58.658872 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:58.662703 14727 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:58.665060 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.666394 14727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:58.666533 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "7e9f403be31740f5bb5f324a5cc118a7"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "cc48be537dcd3a97bc3a554a05563893"
server_key_iv: "2cfd88c49cf08f7f8d6bbb028142dd45"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.667070 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:58.692517 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:58.695613 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:58.695817 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:58.696408 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:58.697480 14770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:46:58.701453 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:46:58.701633 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 00:46:58.701715 14727 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:46:58.703346 14727 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:46:58.703414 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:46:58.703706 14770 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap starting.
I20260927 00:46:58.713536 14727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38179
I20260927 00:46:58.713580 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38179 every 8 connection(s)
I20260927 00:46:58.714483 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:58.715411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14727
I20260927 00:46:58.715816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38525
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=43415
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:58.725701 14770 log.cc:824] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:58.731966 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.717544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60365 (local address 127.0.98.62:46323)
0927 00:46:58.717968 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.717987 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.718681 (+ 694us) server_negotiation.cc:407] Connection header received
0927 00:46:58.719802 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.719821 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.720115 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.720431 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.721521 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.722542 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.723698 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.724153 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.724329 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.726879 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.726886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.726901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.726944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.728738 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.729270 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.729280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.729284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.729401 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.729753 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.729758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.729760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.730191 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.730336 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.730848 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.731102 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260927 00:46:58.731986 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.717522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:60365)
0927 00:46:58.718527 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.718566 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.719353 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.720496 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.720504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.720910 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:58.721350 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.721363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.722707 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.722710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.723563 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.723572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.724304 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.724308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.724514 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.725138 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:58.725154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:58.726688 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:58.728873 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.728878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.728891 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.729150 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.729517 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.729520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.729522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.729658 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.730307 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:58.730313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:58.730556 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.730740 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.731083 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":147,"threads_started":1}
I20260927 00:46:58.734411 14878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:58.734642 14878 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:58.735118 14878 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:58.735239 14770 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:58.735633 14770 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap complete.
I20260927 00:46:58.737071 14770 ts_tablet_manager.cc:1397] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.004s
I20260927 00:46:58.737483 14689 ts_manager.cc:194] Registered new tserver with Master: 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:58.740003 14770 raft_consensus.cc:348] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.740545 14770 raft_consensus.cc:729] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9f403be31740f5bb5f324a5cc118a7, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.741029 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60365
I20260927 00:46:58.741083 14770 consensus_queue.cc:260] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.742205 14770 ts_tablet_manager.cc:1428] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260927 00:46:58.743839 14878 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:58.751623 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.740145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.0.1:46600)
0927 00:46:58.741657 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.741699 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.741878 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.742677 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.742683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.743108 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:58.743244 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.743252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.744522 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.744529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.745185 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.745194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.745928 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.745942 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.746020 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.746558 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:58.746569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:58.747862 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:58.749970 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.749979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.749983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.750212 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.750543 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.750546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.750547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.750638 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.751113 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:58.751118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:58.751219 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.751334 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.751491 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":108,"threads_started":1}
I20260927 00:46:58.751720 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.741043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.0.98.1:38179)
0927 00:46:58.742153 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.742164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.742184 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:46:58.742242 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.742246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.742493 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.742616 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.743377 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.744439 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.745314 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.745824 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.745937 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.747992 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.748002 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.748013 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.748038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.749859 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.750322 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.750328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.750330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.750423 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.750731 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.750735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.750736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.750995 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.751130 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.751429 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.751567 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":95,"threads_started":1}
I20260927 00:46:58.752857 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46600
I20260927 00:46:58.753842 14887 tablet_replica.cc:330] stopping tablet replica
I20260927 00:46:58.754151 14887 raft_consensus.cc:2229] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:58.754397 14887 raft_consensus.cc:2258] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:58.755342 14887 ts_tablet_manager.cc:1905] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:58.758864 14887 ts_tablet_manager.cc:1918] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:58.758989 14887 log.cc:1196] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/7ea22c1cb4e84c6c8ca6ad0015023453
I20260927 00:46:58.759251 14887 ts_tablet_manager.cc:1939] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 7e9f403be31740f5bb5f324a5cc118a7: Deleting consensus metadata
I20260927 00:46:58.760843 14673 catalog_manager.cc:4841] TS 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179): tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (table test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]) successfully deleted
W20260927 00:46:58.837105 14880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:58.837361 14880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:58.837402 14880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:58.841679 14880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:58.841735 14880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:58.841858 14880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:58.847388 14880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43415
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:58.848480 14880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:58.849416 14880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:58.858903 14895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.858949 14880 server_base.cc:1034] running on GCE node
W20260927 00:46:58.858897 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:58.858897 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:58.859619 14880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:58.860286 14880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:58.861455 14880 hybrid_clock.cc:648] HybridClock initialized: now 1790470018861437 us; error 28 us; skew 500 ppm
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:58.864405 14880 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:58.865335 14880 webserver.cc:460] Webserver started at http://127.0.98.2:43415/ using document root <none> and password file <none>
I20260927 00:46:58.865957 14880 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:58.866045 14880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:58.869949 14880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:46:58.872356 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:58.873678 14880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:58.873842 14880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "48c640317f17476aae0cc25a1c5a36b4"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "c8bc77b1bd7169a3567713971c42a033"
server_key_iv: "ac2029f67420080cd795803c7b2c6691"
server_key_version: "encryptionkey@0"
I20260927 00:46:58.874424 14880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:58.905542 14880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:58.909466 14880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:58.909708 14880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:58.910393 14880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:58.911581 14909 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:46:58.915868 14880 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:46:58.915977 14880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:46:58.916061 14880 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:46:58.917829 14880 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:46:58.917912 14880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:58.918216 14909 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Bootstrap starting.
I20260927 00:46:58.924342 14909 log.cc:824] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:58.928126 14880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38525
I20260927 00:46:58.928196 15017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38525 every 8 connection(s)
I20260927 00:46:58.928954 14880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:58.932199 14909 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:58.932621 14909 tablet_bootstrap.cc:492] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Bootstrap complete.
I20260927 00:46:58.932964 14909 ts_tablet_manager.cc:1397] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260927 00:46:58.933929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 14880
I20260927 00:46:58.934396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44553
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=35061
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:58.935748 14909 raft_consensus.cc:348] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.936239 14909 raft_consensus.cc:729] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c640317f17476aae0cc25a1c5a36b4, State: Initialized, Role: FOLLOWER
I20260927 00:46:58.936817 14909 consensus_queue.cc:260] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:58.937808 14909 ts_tablet_manager.cc:1428] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:58.946694 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.930826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56129 (local address 127.0.98.62:46323)
0927 00:46:58.930992 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.930997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.931602 (+ 605us) server_negotiation.cc:407] Connection header received
0927 00:46:58.932580 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.932584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.932645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.932748 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.934326 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.935274 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.938103 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.938371 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.938460 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.942203 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.942209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.942215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.942256 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.944242 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.944764 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.944771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.944774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.944882 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.945237 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.945244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.945247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.945557 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.945682 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.946256 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.946436 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:46:58.947275 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.930947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:56129)
0927 00:46:58.931435 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.931486 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.932378 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.933020 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.933028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.933484 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:58.934153 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.934169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.935445 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.935449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.937958 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.937974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.938500 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.938507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.938774 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.939870 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:58.939899 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:58.941911 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:58.944367 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.944372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.944382 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.944639 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.944979 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.944981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.944983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.945125 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.945682 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:58.945686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:58.945978 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.946138 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.946380 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20260927 00:46:58.948423 15018 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:58.948674 15018 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:58.949143 15018 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:58.950779 14689 ts_manager.cc:194] Registered new tserver with Master: 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
I20260927 00:46:58.952100 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56129
I20260927 00:46:58.954500 15018 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
Sep 27 00:46:58 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:46:58.961169 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.951757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38525 (local address 127.0.0.1:41640)
0927 00:46:58.952409 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.952425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.952509 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.952917 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.952920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.953114 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:58.953200 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.953205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.954224 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.954229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.954915 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.954929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.955705 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.955720 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.955832 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.956288 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:58.956304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:58.957669 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:58.959584 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.959589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.959591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.959789 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.960092 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:58.960095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:58.960097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:58.960183 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:58.960631 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:58.960634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:58.960728 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.960848 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.960991 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570}
I20260927 00:46:58.961233 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.951821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.98.2:38525)
0927 00:46:58.952097 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.952106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.952486 (+ 380us) server_negotiation.cc:407] Connection header received
0927 00:46:58.952615 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.952623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.952790 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.952930 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:58.953308 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.954110 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.955022 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.955564 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.955685 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.957830 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:58.957841 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:58.957852 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:58.957872 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:58.959459 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.959915 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.959920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.959921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.959993 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:58.960272 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:58.960276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:58.960278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:58.960510 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:58.960654 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.960928 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.961082 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260927 00:46:58.962332 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea22c1cb4e84c6c8ca6ad0015023453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41640
I20260927 00:46:58.963411 15024 tablet_replica.cc:330] stopping tablet replica
I20260927 00:46:58.963696 15024 raft_consensus.cc:2229] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:58.963963 15024 raft_consensus.cc:2258] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:58.964967 15024 ts_tablet_manager.cc:1905] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:58.968535 15024 ts_tablet_manager.cc:1918] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:58.968657 15024 log.cc:1196] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/7ea22c1cb4e84c6c8ca6ad0015023453
I20260927 00:46:58.968911 15024 ts_tablet_manager.cc:1939] T 7ea22c1cb4e84c6c8ca6ad0015023453 P 48c640317f17476aae0cc25a1c5a36b4: Deleting consensus metadata
I20260927 00:46:58.970436 14673 catalog_manager.cc:4841] TS 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): tablet 7ea22c1cb4e84c6c8ca6ad0015023453 (table test-table [id=2b3cbc55d7a64ad28ecb7a52f7b1c660]) successfully deleted
W20260927 00:46:59.056046 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:59.056314 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:59.056381 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:59.060555 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:59.060609 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:59.060665 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:59.065950 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35061
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:59.066922 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:59.067854 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:59.077868 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.077877 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.077877 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.078270 15021 server_base.cc:1034] running on GCE node
I20260927 00:46:59.078609 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:59.079285 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:59.080474 15021 hybrid_clock.cc:648] HybridClock initialized: now 1790470019080444 us; error 47 us; skew 500 ppm
Sep 27 00:46:59 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:59.083305 15021 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:59.084128 15021 webserver.cc:460] Webserver started at http://127.0.98.3:35061/ using document root <none> and password file <none>
I20260927 00:46:59.084641 15021 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:59.084712 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:59.088330 15021 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:46:59.090309 15039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.091566 15021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:59.091655 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "af64511fa3a242b5947eb4ddc59465f8"
format_stamp: "Formatted at 2026-09-27 00:46:58 on dist-test-slave-l5sb"
server_key: "92b7ee1ad0e4202d14616cbe07f15f53"
server_key_iv: "3645ee616c8b920237588d0bb5a0f171"
server_key_version: "encryptionkey@0"
I20260927 00:46:59.092156 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:59.123157 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:59.125976 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:59.126132 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:59.126641 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:59.127384 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:59.127429 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.127467 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:59.127486 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.136283 15021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44553
I20260927 00:46:59.136314 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44553 every 8 connection(s)
I20260927 00:46:59.137245 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:59.142141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15021
Sep 27 00:46:59 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:59.151201 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.142799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.98.62:46323)
0927 00:46:59.143058 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.143062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.143226 (+ 164us) server_negotiation.cc:407] Connection header received
0927 00:46:59.143325 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.143328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.143371 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.143449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:59.144336 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.144916 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.146021 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.146326 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.146401 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.147744 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:59.147748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:59.147751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:59.147783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:59.149486 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.149957 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.149961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.149963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.150044 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.150312 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.150317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.150319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.150532 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:59.150742 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.150895 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.151020 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260927 00:46:59.151922 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.138910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41493 (local address 127.0.98.62:46323)
0927 00:46:59.139051 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.139055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.139806 (+ 751us) server_negotiation.cc:407] Connection header received
0927 00:46:59.141207 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.141210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.141270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.141376 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:59.142735 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.143456 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.145032 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.145281 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.145355 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.148072 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:59.148075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:59.148078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:59.148119 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:59.149569 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.150133 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.150140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.150142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.150223 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.150564 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.150567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.150569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.150737 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:59.150837 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.151713 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.151826 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:46:59.152597 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.139241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:41493)
0927 00:46:59.139661 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.139701 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:59.140992 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:59.141558 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:59.141565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:59.142022 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:59.142589 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.142616 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.143590 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.143594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.144907 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.144916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.145423 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.145428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.145625 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.146277 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:59.146294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:59.147902 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:59.149690 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:59.149695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:59.149709 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:59.150033 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:59.150321 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:59.150324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:59.150326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:59.150460 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:59.150838 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:59.150845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:59.151114 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 00:46:59.151338 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 00:46:59.151604 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260927 00:46:59.153736 15153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46323
I20260927 00:46:59.153996 15153 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:59.154462 15153 heartbeater.cc:511] Master 127.0.98.62:46323 requested a full tablet report, sending...
I20260927 00:46:59.155388 14689 ts_manager.cc:194] Registered new tserver with Master: af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553)
I20260927 00:46:59.155926 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41493
I20260927 00:46:59.164740 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:59.168651 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.165338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.0.98.62:46323)
0927 00:46:59.165454 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.165458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.165612 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:46:59.165803 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.165806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.165911 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.165990 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:59.166159 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.166637 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.167434 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.167676 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.167750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.168025 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:59.168369 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:59.168454 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.168508 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.168557 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:46:59.169229 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:59.172582 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:59.179832 14812 tablet_service.cc:1467] Processing CreateTablet for tablet 73e4b7919409455ba9a846bded8be5cc (DEFAULT_TABLE table=test-table [id=bb645dc8ce854801ae4dd5aa5e067ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.179862 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 73e4b7919409455ba9a846bded8be5cc (DEFAULT_TABLE table=test-table [id=bb645dc8ce854801ae4dd5aa5e067ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.180423 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e4b7919409455ba9a846bded8be5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.180449 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e4b7919409455ba9a846bded8be5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.182950 14770 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap starting.
I20260927 00:46:59.183202 14909 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Bootstrap starting.
I20260927 00:46:59.183770 14770 tablet_bootstrap.cc:654] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.184067 14909 tablet_bootstrap.cc:654] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.185216 14770 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: No bootstrap required, opened a new log
I20260927 00:46:59.185299 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.178947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44553 (local address 127.0.0.1:59244)
0927 00:46:59.179083 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.179099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:59.179192 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:59.179926 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:59.179929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:59.179948 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:59.180033 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.180039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.181565 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.181569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.182795 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.182806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.184818 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.184822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.184934 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.184984 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:46:59.185043 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 00:46:59.185144 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:46:59.185320 14770 ts_tablet_manager.cc:1397] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:59.185529 14909 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: No bootstrap required, opened a new log
I20260927 00:46:59.185626 14909 ts_tablet_manager.cc:1397] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:59.185882 14770 raft_consensus.cc:348] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.185952 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.179137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.0.98.3:44553)
0927 00:46:59.179455 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.179467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.179491 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:46:59.179563 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.179569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.179778 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.179941 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:59.180184 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.181401 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.182942 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.184668 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.185477 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.185632 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.185689 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.185820 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260927 00:46:59.186007 14770 raft_consensus.cc:374] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.186054 14770 raft_consensus.cc:729] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9f403be31740f5bb5f324a5cc118a7, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.186094 14909 raft_consensus.cc:348] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.186213 14909 raft_consensus.cc:374] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.186249 14909 raft_consensus.cc:729] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c640317f17476aae0cc25a1c5a36b4, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.186204 14770 consensus_queue.cc:260] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.186367 14909 consensus_queue.cc:260] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.186547 14770 ts_tablet_manager.cc:1428] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:59.186602 14909 ts_tablet_manager.cc:1428] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:46:59.189025 15087 tablet_service.cc:1467] Processing CreateTablet for tablet 73e4b7919409455ba9a846bded8be5cc (DEFAULT_TABLE table=test-table [id=bb645dc8ce854801ae4dd5aa5e067ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.190110 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e4b7919409455ba9a846bded8be5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.196318 15165 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Bootstrap starting.
I20260927 00:46:59.198279 15165 tablet_bootstrap.cc:654] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.199396 15165 log.cc:824] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:59.201148 15165 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: No bootstrap required, opened a new log
I20260927 00:46:59.201351 15165 ts_tablet_manager.cc:1397] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:46:59.204244 15165 raft_consensus.cc:348] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.204550 15165 raft_consensus.cc:374] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.204610 15165 raft_consensus.cc:729] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af64511fa3a242b5947eb4ddc59465f8, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.204982 15165 consensus_queue.cc:260] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.205585 15153 heartbeater.cc:503] Master 127.0.98.62:46323 was elected leader, sending a full tablet report...
I20260927 00:46:59.205943 15165 ts_tablet_manager.cc:1428] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 00:46:59.215689 14879 tablet.cc:2369] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:59.254247 15022 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:59.254464 15022 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.255666 15022 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:59.260442 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.255879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44553 (local address 127.0.98.2:37541)
0927 00:46:59.256485 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.256507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:59.256628 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:59.256861 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:59.256864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:59.256893 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:59.257028 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.257035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.258039 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.258045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.259182 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.259191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.259943 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.259949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.260071 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.260121 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:46:59.260199 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:46:59.260276 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":346,"thread_start_us":108,"threads_started":1}
I20260927 00:46:59.260444 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.255990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37541 (local address 127.0.98.3:44553)
0927 00:46:59.256150 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.256157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.256589 (+ 432us) server_negotiation.cc:407] Connection header received
0927 00:46:59.256718 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.256721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.256769 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.256850 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:59.257150 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.257910 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.259304 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.259798 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.259883 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.259935 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.260232 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.260321 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:46:59.260468 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.255879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.98.2:40245)
0927 00:46:59.256031 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.256047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:59.256165 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:59.256579 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:59.256582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:59.256600 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:59.256734 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.256739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.257847 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.257850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.259292 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:59.259302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.260069 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.260072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:59.260158 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.260204 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:46:59.260289 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:46:59.260353 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":4,"spinlock_wait_cycles":10496}
I20260927 00:46:59.260522 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.256008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40245 (local address 127.0.98.1:38179)
0927 00:46:59.256345 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.256348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.256359 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:46:59.256404 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.256406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.256454 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.256568 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:59.256846 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.257689 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.259417 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.259911 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.260023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.260199 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.260306 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.260377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260927 00:46:59.261078 14832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e4b7919409455ba9a846bded8be5cc" candidate_uuid: "48c640317f17476aae0cc25a1c5a36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9f403be31740f5bb5f324a5cc118a7" is_pre_election: true
I20260927 00:46:59.261094 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e4b7919409455ba9a846bded8be5cc" candidate_uuid: "48c640317f17476aae0cc25a1c5a36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af64511fa3a242b5947eb4ddc59465f8" is_pre_election: true
I20260927 00:46:59.261395 14832 raft_consensus.cc:2454] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c640317f17476aae0cc25a1c5a36b4 in term 0.
I20260927 00:46:59.261433 15107 raft_consensus.cc:2454] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c640317f17476aae0cc25a1c5a36b4 in term 0.
I20260927 00:46:59.261909 14903 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48c640317f17476aae0cc25a1c5a36b4, 7e9f403be31740f5bb5f324a5cc118a7, af64511fa3a242b5947eb4ddc59465f8; no voters:
I20260927 00:46:59.262197 15022 raft_consensus.cc:2781] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:59.262269 15022 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:59.262300 15022 raft_consensus.cc:3037] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:59.263337 15022 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.263737 15022 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [CANDIDATE]: Term 1 election: Requested vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179)
I20260927 00:46:59.264047 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e4b7919409455ba9a846bded8be5cc" candidate_uuid: "48c640317f17476aae0cc25a1c5a36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af64511fa3a242b5947eb4ddc59465f8"
I20260927 00:46:59.264123 14832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e4b7919409455ba9a846bded8be5cc" candidate_uuid: "48c640317f17476aae0cc25a1c5a36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9f403be31740f5bb5f324a5cc118a7"
I20260927 00:46:59.264168 15107 raft_consensus.cc:3037] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:59.264278 14832 raft_consensus.cc:3037] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:59.265249 15107 raft_consensus.cc:2454] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c640317f17476aae0cc25a1c5a36b4 in term 1.
I20260927 00:46:59.265448 14832 raft_consensus.cc:2454] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c640317f17476aae0cc25a1c5a36b4 in term 1.
I20260927 00:46:59.265542 14905 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8; no voters:
I20260927 00:46:59.265726 15022 raft_consensus.cc:2781] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:59.265974 15022 raft_consensus.cc:686] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 1 LEADER]: Becoming Leader. State: Replica: 48c640317f17476aae0cc25a1c5a36b4, State: Running, Role: LEADER
I20260927 00:46:59.266268 15022 consensus_queue.cc:237] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.269083 14687 catalog_manager.cc:5495] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "48c640317f17476aae0cc25a1c5a36b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } health_report { overall_health: UNKNOWN } } }
I20260927 00:46:59.285995 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.282456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.2:38525)
0927 00:46:59.282598 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.282605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.282619 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:46:59.282798 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.282800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.282852 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.282957 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:59.283199 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.283727 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.284575 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.284809 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.284917 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.285184 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:59.285592 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:59.285712 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.285804 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.285859 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:46:59.289724 15107 raft_consensus.cc:1261] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Refusing update from remote peer 48c640317f17476aae0cc25a1c5a36b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:59.289721 14832 raft_consensus.cc:1261] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Refusing update from remote peer 48c640317f17476aae0cc25a1c5a36b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:59.290448 15170 consensus_queue.cc:1035] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:59.290627 15022 consensus_queue.cc:1035] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:59.298074 15175 mvcc.cc:205] Tried to move back new op lower bound from 7333765199004364800 to 7333765198917070848. Current Snapshot: MvccSnapshot[applied={T|T < 7333765199004364800}]
I20260927 00:46:59.306696 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:59.306831 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:59.309197 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Sending DeleteTablet for 3 replicas of tablet 73e4b7919409455ba9a846bded8be5cc
I20260927 00:46:59.309891 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e4b7919409455ba9a846bded8be5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46600
I20260927 00:46:59.309942 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e4b7919409455ba9a846bded8be5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59244
I20260927 00:46:59.310048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14727
I20260927 00:46:59.310940 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e4b7919409455ba9a846bded8be5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41640
I20260927 00:46:59.311129 15024 tablet_replica.cc:330] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: stopping tablet replica
I20260927 00:46:59.311266 15024 raft_consensus.cc:2229] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:59.311555 15024 raft_consensus.cc:2258] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:59.312525 15024 ts_tablet_manager.cc:1905] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:59.315766 15024 ts_tablet_manager.cc:1918] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:59.315853 15024 log.cc:1196] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/73e4b7919409455ba9a846bded8be5cc
I20260927 00:46:59.316123 15024 ts_tablet_manager.cc:1939] T 73e4b7919409455ba9a846bded8be5cc P 48c640317f17476aae0cc25a1c5a36b4: Deleting consensus metadata
I20260927 00:46:59.316149 15183 tablet_replica.cc:330] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: stopping tablet replica
I20260927 00:46:59.316699 15183 raft_consensus.cc:2229] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:59.317054 15183 raft_consensus.cc:2258] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:59.317147 14673 catalog_manager.cc:4841] TS 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): tablet 73e4b7919409455ba9a846bded8be5cc (table test-table [id=bb645dc8ce854801ae4dd5aa5e067ec4]) successfully deleted
W20260927 00:46:59.317584 14673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38179 (error 108)
I20260927 00:46:59.318643 15183 ts_tablet_manager.cc:1905] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:59.318661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14880
I20260927 00:46:59.318701 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.318415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38179 (local address 127.0.0.1:46610)
0927 00:46:59.318541 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.318616 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:46:59.318845 14673 catalog_manager.cc:4584] TS 7e9f403be31740f5bb5f324a5cc118a7 (127.0.98.1:38179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73e4b7919409455ba9a846bded8be5cc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38179: connect: Connection refused (error 111)
I20260927 00:46:59.321429 15183 ts_tablet_manager.cc:1918] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:59.321557 15183 log.cc:1196] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/73e4b7919409455ba9a846bded8be5cc
I20260927 00:46:59.321949 15183 ts_tablet_manager.cc:1939] T 73e4b7919409455ba9a846bded8be5cc P af64511fa3a242b5947eb4ddc59465f8: Deleting consensus metadata
I20260927 00:46:59.323244 14675 catalog_manager.cc:4841] TS af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): tablet 73e4b7919409455ba9a846bded8be5cc (table test-table [id=bb645dc8ce854801ae4dd5aa5e067ec4]) successfully deleted
I20260927 00:46:59.325760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15021
I20260927 00:46:59.331128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 14656
I20260927 00:46:59.337357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--webserver_interface=127.0.98.62
--webserver_port=41829
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:59.444705 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:59.445003 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:59.445091 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:59.449083 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:59.449141 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:59.449188 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:59.449224 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:59.449255 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:59.454403 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:59.455446 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:59.456462 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:59.464893 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.465080 15185 server_base.cc:1034] running on GCE node
W20260927 00:46:59.464893 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.464895 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.465696 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:59.466792 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:59.467983 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790470019467964 us; error 30 us; skew 500 ppm
Sep 27 00:46:59 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:59.470692 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:59.471483 15185 webserver.cc:460] Webserver started at http://127.0.98.62:41829/ using document root <none> and password file <none>
I20260927 00:46:59.471992 15185 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:59.472052 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:59.475670 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:59.477586 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.478852 15185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:59.478973 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "edcf9d6c1c5541b6b965b1f270fc55fd"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "4d95516476c786c32c91ce1180a02b1b"
server_key_iv: "1866bc6396899d2d2af3364e4ca6f677"
server_key_version: "encryptionkey@0"
I20260927 00:46:59.479490 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:59.496590 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:59.499287 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:59.499485 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:59.505976 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46323
I20260927 00:46:59.506248 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46323 every 8 connection(s)
I20260927 00:46:59.507016 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:59.511582 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:46:59.513020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15185
I20260927 00:46:59.513228 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap starting.
I20260927 00:46:59.513477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38179
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=46695
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:59.537122 15253 log.cc:824] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:59.549172 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:59.549544 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd: Bootstrap complete.
I20260927 00:46:59.553416 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:59.553865 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.554352 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:59.554466 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:59.554556 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:59.554665 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:46:59.556471 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:59.556799 15253 leader_election.cc:302] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edcf9d6c1c5541b6b965b1f270fc55fd; no voters:
I20260927 00:46:59.557128 15253 leader_election.cc:288] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:46:59.557250 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:46:59.557482 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [term 3 LEADER]: Becoming Leader. State: Replica: edcf9d6c1c5541b6b965b1f270fc55fd, State: Running, Role: LEADER
I20260927 00:46:59.557814 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } }
I20260927 00:46:59.558394 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:59.559150 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:59.559271 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:59.559590 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader edcf9d6c1c5541b6b965b1f270fc55fd. Latest consensus state: current_term: 3 leader_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edcf9d6c1c5541b6b965b1f270fc55fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46323 } } }
I20260927 00:46:59.559672 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P edcf9d6c1c5541b6b965b1f270fc55fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:59.560009 15266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:59.564882 15266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:59.565577 15266 catalog_manager.cc:1259] Loaded cluster ID: 04d809aeb5a24b0c916906c13b26c431
I20260927 00:46:59.565639 15266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:46:59.566926 15266 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdbcf7259ad google::LogMessage::Fail() at ??:0
@ 0x7fdbcf72888a google::LogMessage::SendToLog() at ??:0
@ 0x7fdbcf7254de google::LogMessage::Flush() at ??:0
@ 0x7fdbcf726a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdbda068fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdbda069d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdbda066579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdbda0aa640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdbd19773d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdbd1977c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdbd19793f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdbd1968512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdbd2fac6db start_thread at ??:0
@ 0x7fdbce22361f clone at ??:0
W20260927 00:46:59.668823 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:59.669204 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:59.669275 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:59.675331 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:59.675412 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:59.675500 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:59.683429 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46695
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:59.684804 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:59.686128 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:59.697583 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.697609 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.697607 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.698185 15255 server_base.cc:1034] running on GCE node
I20260927 00:46:59.698570 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:59.699237 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:59.700421 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790470019700394 us; error 45 us; skew 500 ppm
Sep 27 00:46:59 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:59.703279 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:59.704219 15255 webserver.cc:460] Webserver started at http://127.0.98.1:46695/ using document root <none> and password file <none>
I20260927 00:46:59.704766 15255 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:59.704826 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:59.708844 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:59.711261 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.712507 15255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:46:59.712635 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "7e9f403be31740f5bb5f324a5cc118a7"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "cc48be537dcd3a97bc3a554a05563893"
server_key_iv: "2cfd88c49cf08f7f8d6bbb028142dd45"
server_key_version: "encryptionkey@0"
I20260927 00:46:59.713181 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:59.742434 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:59.745961 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:59.746172 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:59.746817 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:59.747939 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:46:59.753408 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:46:59.753481 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 00:46:59.753556 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:46:59.755218 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:46:59.755287 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:46:59.755579 15311 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap starting.
I20260927 00:46:59.765686 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38179
I20260927 00:46:59.765745 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38179 every 8 connection(s)
I20260927 00:46:59.766637 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:59.771153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15255
I20260927 00:46:59.771574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38525
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=43415
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:46:59.777724 15311 log.cc:824] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:59.784116 15311 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:46:59.784468 15311 tablet_bootstrap.cc:492] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Bootstrap complete.
I20260927 00:46:59.785595 15311 ts_tablet_manager.cc:1397] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20260927 00:46:59.787847 15311 raft_consensus.cc:348] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.788262 15311 raft_consensus.cc:729] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9f403be31740f5bb5f324a5cc118a7, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.788776 15311 consensus_queue.cc:260] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:46:59.789681 15311 ts_tablet_manager.cc:1428] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260927 00:46:59.820533 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:46:59.768474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:33007)
0927 00:46:59.768931 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.768984 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:59.769808 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:59.819819 (+ 50011us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
W20260927 00:46:59.821002 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46323: Client connection negotiation failed: client connection to 127.0.98.62:46323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:46:59.821744 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.821507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:59159)
0927 00:46:59.821632 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.821691 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:46:59.822440 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.822235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:43363)
0927 00:46:59.822343 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.822392 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:46:59.822633 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:46:59.894610 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:59.894891 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:59.894963 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:59.899075 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:59.899158 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:59.899246 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:59.904456 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43415
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:46:59.905510 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:59.906543 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:59.916121 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:59.916131 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.916244 15422 server_base.cc:1034] running on GCE node
W20260927 00:46:59.916121 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:59.916939 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:59.917606 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:59.918785 15422 hybrid_clock.cc:648] HybridClock initialized: now 1790470019918768 us; error 38 us; skew 500 ppm
Sep 27 00:46:59 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:59.921351 15422 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:59.922259 15422 webserver.cc:460] Webserver started at http://127.0.98.2:43415/ using document root <none> and password file <none>
I20260927 00:46:59.922802 15422 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:59.922890 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:59.926558 15422 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:46:59.928485 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.929644 15422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:59.929805 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "48c640317f17476aae0cc25a1c5a36b4"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-l5sb"
server_key: "c8bc77b1bd7169a3567713971c42a033"
server_key_iv: "ac2029f67420080cd795803c7b2c6691"
server_key_version: "encryptionkey@0"
I20260927 00:46:59.930356 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:59.946285 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:59.949292 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:59.949488 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:59.950080 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:59.950860 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:59.950932 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.951023 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:59.951071 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:59.959861 15422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38525
I20260927 00:46:59.959889 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38525 every 8 connection(s)
I20260927 00:46:59.960821 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:59.964474 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.962801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:55705)
0927 00:46:59.963238 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.963361 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
W20260927 00:46:59.964903 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46323: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
I20260927 00:46:59.965499 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.965277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:58577)
0927 00:46:59.965389 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.965435 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:46:59.966109 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.965902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:32943)
0927 00:46:59.966001 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:59.966047 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:46:59.966267 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:46:59.968279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15422
I20260927 00:46:59.968607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44553
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46323
--webserver_port=35061
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:00.080839 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:00.081110 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:00.081152 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:00.085283 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:00.085345 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:00.085409 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:00.090603 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35061
--tserver_master_addrs=127.0.98.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:00.091629 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:00.092639 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:00.102205 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:00.102167 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:00.102205 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:00.102350 15556 server_base.cc:1034] running on GCE node
I20260927 00:47:00.102857 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:00.103492 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:00.104655 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790470020104638 us; error 27 us; skew 500 ppm
Sep 27 00:47:00 dist-test-slave-l5sb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:00.107333 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:00.108224 15556 webserver.cc:460] Webserver started at http://127.0.98.3:35061/ using document root <none> and password file <none>
I20260927 00:47:00.108731 15556 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:00.108786 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:00.112442 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:47:00.114347 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:00.115510 15556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:00.115634 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "af64511fa3a242b5947eb4ddc59465f8"
format_stamp: "Formatted at 2026-09-27 00:46:58 on dist-test-slave-l5sb"
server_key: "92b7ee1ad0e4202d14616cbe07f15f53"
server_key_iv: "3645ee616c8b920237588d0bb5a0f171"
server_key_version: "encryptionkey@0"
I20260927 00:47:00.116134 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:00.132884 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:00.136162 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:00.136363 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:00.136950 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:00.137760 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:00.137880 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:00.137950 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:00.137996 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:00.146837 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44553
I20260927 00:47:00.146867 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44553 every 8 connection(s)
I20260927 00:47:00.147815 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:00.151504 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.149854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:60419)
0927 00:47:00.150284 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:00.150381 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
W20260927 00:47:00.151932 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46323: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
I20260927 00:47:00.152468 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.152279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:55735)
0927 00:47:00.152374 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:00.152419 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 00:47:00.153046 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.152871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:57303)
0927 00:47:00.152960 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:00.153001 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:47:00.153244 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:47:00.154996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15556
I20260927 00:47:00.823657 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.823228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:41121)
0927 00:47:00.823482 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:00.823546 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260927 00:47:00.967165 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.966771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:35683)
0927 00:47:00.967016 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:00.967071 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 00:47:01.154191 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.153806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:46121)
0927 00:47:01.154047 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.154096 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260927 00:47:01.522852 15692 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:01.523092 15692 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:47:01.524232 15692 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
Sep 27 00:47:01 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 00:47:01 dist-test-slave-l5sb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 00:47:01.538990 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.524493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38525 (local address 127.0.98.1:40677)
0927 00:47:01.524817 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.524832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:01.524945 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:01.526562 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:01.526568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:01.527118 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:01.527567 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:01.527581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.528845 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.528848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:01.529946 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:01.529956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.530654 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:01.530904 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:01.531565 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:01.531581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:01.533222 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:01.536006 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:01.536008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:01.536011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:01.536463 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:01.536869 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:01.536872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:01.536874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:01.537070 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:01.537841 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:01.537848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:01.538180 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 00:47:01.538399 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:47:01.538821 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":441,"spinlock_wait_cycles":34048,"thread_start_us":60,"threads_started":1}
I20260927 00:47:01.539026 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.524713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40677 (local address 127.0.98.2:38525)
0927 00:47:01.525085 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:01.525096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:01.525130 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:47:01.525830 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:01.525839 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:01.526207 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:01.526500 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:01.527742 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.528692 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530072 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.530527 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530704 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:01.533511 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:01.533518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:01.533551 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:01.533587 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:01.535860 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:01.536587 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:01.536593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:01.536599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:01.536759 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:01.537171 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:01.537174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:01.537176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:01.537697 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:01.537859 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:01.538555 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 00:47:01.538839 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":64,"threads_started":1}
I20260927 00:47:01.538988 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.524422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44553 (local address 127.0.98.1:60025)
0927 00:47:01.524720 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.524734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:01.524843 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:01.526562 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:01.526568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:01.527119 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:01.527566 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:01.527580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.528835 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.528840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:01.529945 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:01.529955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.530655 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:01.530901 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:01.531619 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:01.531631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:01.533713 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:01.535990 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:01.535997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:01.536011 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:01.536463 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:01.536869 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:01.536872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:01.536874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:01.537070 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:01.537841 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:01.537848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:01.538180 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 00:47:01.538399 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:47:01.538770 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":206,"spinlock_wait_cycles":25344,"thread_start_us":85,"threads_started":1}
I20260927 00:47:01.539654 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.524597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60025 (local address 127.0.98.3:44553)
0927 00:47:01.525075 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:01.525095 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:01.525129 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:47:01.525833 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:01.525839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:01.526207 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:01.526500 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:01.527728 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.528691 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530065 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:01.530526 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:01.530702 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:01.533924 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:01.533933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:01.533952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:01.533988 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:01.535860 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:01.536587 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:01.536593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:01.536599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:01.536759 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:01.537171 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:01.537174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:01.537176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:01.537697 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:01.537858 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:01.538547 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 00:47:01.538839 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":124,"threads_started":1}
W20260927 00:47:01.541761 15307 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
W20260927 00:47:01.541960 15305 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
I20260927 00:47:01.542065 15305 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9f403be31740f5bb5f324a5cc118a7; no voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8
I20260927 00:47:01.542291 15692 raft_consensus.cc:2726] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:47:01.824745 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.824429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:48139)
0927 00:47:01.824591 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.824658 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 00:47:01.968343 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.967891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:37997)
0927 00:47:01.968171 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.968233 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20260927 00:47:02.155543 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:02.155035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:51271)
0927 00:47:02.155335 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:02.155419 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260927 00:47:02.826004 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:02.825546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:57119)
0927 00:47:02.825829 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:02.825884 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260927 00:47:02.969551 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:02.969117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:33123)
0927 00:47:02.969392 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:02.969452 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260927 00:47:03.156698 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.156299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:52497)
0927 00:47:03.156549 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:03.156608 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260927 00:47:03.756520 15702 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:03.756687 15702 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:47:03.757135 15702 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
W20260927 00:47:03.757921 15305 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
W20260927 00:47:03.758024 15307 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
I20260927 00:47:03.758142 15307 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9f403be31740f5bb5f324a5cc118a7; no voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8
I20260927 00:47:03.758337 15702 raft_consensus.cc:2726] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:47:03.827132 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.826700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:54753)
0927 00:47:03.826982 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:03.827038 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260927 00:47:03.970727 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.970329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:60091)
0927 00:47:03.970579 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:03.970637 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260927 00:47:04.157925 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.157456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:45207)
0927 00:47:04.157715 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:04.157817 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260927 00:47:04.828408 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.827936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:55845)
0927 00:47:04.828252 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:04.828311 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260927 00:47:04.971941 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.971518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:34019)
0927 00:47:04.971786 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:04.971848 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260927 00:47:05.159056 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:05.158653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:53701)
0927 00:47:05.158904 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:05.158963 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260927 00:47:05.829576 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:05.829147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:39227)
0927 00:47:05.829409 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:05.829479 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260927 00:47:05.973306 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:05.972749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:46671)
0927 00:47:05.973124 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:05.973194 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260927 00:47:06.160197 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:06.159759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:45095)
0927 00:47:06.160025 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:06.160093 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260927 00:47:06.615507 15712 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:06.615669 15712 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:47:06.616124 15712 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
W20260927 00:47:06.616873 15307 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
W20260927 00:47:06.616962 15305 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
I20260927 00:47:06.617070 15305 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9f403be31740f5bb5f324a5cc118a7; no voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8
I20260927 00:47:06.617202 15712 raft_consensus.cc:2726] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:47:06.830710 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:06.830310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:35789)
0927 00:47:06.830558 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:06.830614 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260927 00:47:06.974608 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:06.974144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:60963)
0927 00:47:06.974455 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:06.974512 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260927 00:47:07.161345 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:07.160917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:47481)
0927 00:47:07.161187 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:07.161240 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":67,"threads_started":1}
I20260927 00:47:07.831966 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:07.831555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:59233)
0927 00:47:07.831818 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:07.831873 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260927 00:47:07.975783 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:07.975366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:47071)
0927 00:47:07.975624 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:07.975687 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260927 00:47:08.162578 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:08.162182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:50621)
0927 00:47:08.162425 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:08.162483 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260927 00:47:08.833109 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:08.832705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:46547)
0927 00:47:08.832953 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:08.833013 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260927 00:47:08.976984 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:08.976581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:37635)
0927 00:47:08.976828 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:08.976882 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260927 00:47:09.163838 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.163314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:58027)
0927 00:47:09.163574 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.163631 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260927 00:47:09.834408 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.834009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:60319)
0927 00:47:09.834258 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.834315 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260927 00:47:09.978241 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.977744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:40929)
0927 00:47:09.978080 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.978143 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":75,"threads_started":1}
I20260927 00:47:10.165120 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:10.164660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:39333)
0927 00:47:10.164961 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:10.165023 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260927 00:47:10.835585 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:10.835169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:52763)
0927 00:47:10.835435 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:10.835490 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260927 00:47:10.979616 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:10.979183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:45037)
0927 00:47:10.979457 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:10.979520 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260927 00:47:11.166531 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.166083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:55767)
0927 00:47:11.166369 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:11.166436 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20260927 00:47:11.503577 15728 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:11.503715 15728 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:47:11.504233 15728 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
W20260927 00:47:11.505048 15307 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
W20260927 00:47:11.505157 15305 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
I20260927 00:47:11.505281 15305 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9f403be31740f5bb5f324a5cc118a7; no voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8
I20260927 00:47:11.505463 15728 raft_consensus.cc:2726] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:47:11.836869 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.836371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:52521)
0927 00:47:11.836686 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:11.836761 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
I20260927 00:47:11.980947 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.980503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:45557)
0927 00:47:11.980769 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:11.980843 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260927 00:47:12.167800 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.167378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:44839)
0927 00:47:12.167626 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:12.167684 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260927 00:47:12.838066 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.837614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:60571)
0927 00:47:12.837916 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:12.837969 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
I20260927 00:47:12.982221 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.981752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:41275)
0927 00:47:12.982067 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:12.982125 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
I20260927 00:47:13.168958 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:13.168535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:56491)
0927 00:47:13.168803 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:13.168864 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260927 00:47:13.839211 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:13.838778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:48953)
0927 00:47:13.839054 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:13.839113 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260927 00:47:13.983459 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:13.983024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:47301)
0927 00:47:13.983299 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:13.983360 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260927 00:47:14.170179 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.169694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:54533)
0927 00:47:14.170028 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.170086 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":143,"threads_started":1}
I20260927 00:47:14.446871 15738 raft_consensus.cc:482] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:14.446998 15738 raft_consensus.cc:504] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af64511fa3a242b5947eb4ddc59465f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44553 } } peers { permanent_uuid: "48c640317f17476aae0cc25a1c5a36b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38525 } } peers { permanent_uuid: "7e9f403be31740f5bb5f324a5cc118a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38179 } }
I20260927 00:47:14.447412 15738 leader_election.cc:288] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553), 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525)
W20260927 00:47:14.448140 15307 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af64511fa3a242b5947eb4ddc59465f8 (127.0.98.3:44553): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
W20260927 00:47:14.448412 15305 leader_election.cc:341] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48c640317f17476aae0cc25a1c5a36b4 (127.0.98.2:38525): Not found: Tablet not found: 73e4b7919409455ba9a846bded8be5cc
I20260927 00:47:14.448498 15305 leader_election.cc:302] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9f403be31740f5bb5f324a5cc118a7; no voters: 48c640317f17476aae0cc25a1c5a36b4, af64511fa3a242b5947eb4ddc59465f8
I20260927 00:47:14.448681 15738 raft_consensus.cc:2726] T 73e4b7919409455ba9a846bded8be5cc P 7e9f403be31740f5bb5f324a5cc118a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:47:14.840516 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.840109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.1:45917)
0927 00:47:14.840366 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.840420 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260927 00:47:14.984692 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.984309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.2:46793)
0927 00:47:14.984546 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.984599 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260927 00:47:15.166064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15255
I20260927 00:47:15.171533 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.171111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46323 (local address 127.0.98.3:54307)
0927 00:47:15.171380 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:15.171438 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260927 00:47:15.171962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15422
I20260927 00:47:15.177496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15556
2026-09-27T00:47:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19200 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:15 dist-test-slave-l5sb krb5kdc[15744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:15 dist-test-slave-l5sb krb5kdc[15744](info): set up 2 sockets
Sep 27 00:47:15 dist-test-slave-l5sb krb5kdc[15744](info): commencing operation
krb5kdc: starting...
W20260927 00:47:16.214099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:16.336615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:16.445709 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:16.445998 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:16.446041 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:16.449976 15757 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:47:16.450032 15757 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:47:16.450048 15757 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:47:16.450062 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:16.450080 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:16.450098 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:16.450114 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:16.450136 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:16.455420 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:16.456394 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:16.457355 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:16.466176 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.466143 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.466135 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.466190 15757 server_base.cc:1034] running on GCE node
I20260927 00:47:16.466846 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:16.467926 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:16.469125 15757 hybrid_clock.cc:648] HybridClock initialized: now 1790470036469107 us; error 26 us; skew 500 ppm
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:16.471846 15757 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:16.472805 15757 webserver.cc:460] Webserver started at http://127.0.98.62:39303/ using document root <none> and password file <none>
I20260927 00:47:16.473368 15757 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:16.473457 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:16.473690 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:16.475781 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "907ec10faa9d4b06800b79e44f2cb520"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "2ef3645f6b9e54749a725765e7f85e39"
server_key_iv: "7be138a1d65ca83af4c8c70583a060c2"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.476356 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "907ec10faa9d4b06800b79e44f2cb520"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "2ef3645f6b9e54749a725765e7f85e39"
server_key_iv: "7be138a1d65ca83af4c8c70583a060c2"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.480173 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:16.482611 15772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.483791 15757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:47:16.483935 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "907ec10faa9d4b06800b79e44f2cb520"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "2ef3645f6b9e54749a725765e7f85e39"
server_key_iv: "7be138a1d65ca83af4c8c70583a060c2"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.484046 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:16.508047 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:16.511142 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:16.511354 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:16.518532 15757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41447
I20260927 00:47:16.518826 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41447 every 8 connection(s)
I20260927 00:47:16.519593 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:16.522320 15825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.522723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15757
I20260927 00:47:16.522859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:16.523145 392 external_mini_cluster.cc:1428] Setting key 04d94e7541b47e5eb0587d4fcdd27413
I20260927 00:47:16.527987 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap starting.
I20260927 00:47:16.530208 15825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.530980 15825 log.cc:824] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:16.533121 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: No bootstrap required, opened a new log
I20260927 00:47:16.537139 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.524531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.0.98.62:41447)
0927 00:47:16.524977 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.524991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.525024 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:47:16.525661 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.525678 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.526071 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.526394 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.527188 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.528168 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.528994 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.529478 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.529649 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.532399 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.532404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.532416 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:16.532441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.534317 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:16.534802 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:16.534807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:16.534812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:16.534918 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:16.535248 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:16.535251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:16.535253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:16.535629 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.535772 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.536074 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.536368 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":133,"threads_started":1}
I20260927 00:47:16.537115 15825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:16.537423 15825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.537496 15825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.537956 15825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:16.538105 15825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:16.538185 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:16.538305 15825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.539175 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:16.539515 15825 leader_election.cc:302] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907ec10faa9d4b06800b79e44f2cb520; no voters:
I20260927 00:47:16.539837 15825 leader_election.cc:288] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:16.539906 15830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:16.540166 15830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 LEADER]: Becoming Leader. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Running, Role: LEADER
I20260927 00:47:16.540570 15830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:16.541004 15825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:16.542459 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:16.542475 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907ec10faa9d4b06800b79e44f2cb520. Latest consensus state: current_term: 1 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:16.542693 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.542723 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.543057 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:16.545646 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:16.550544 15844 catalog_manager.cc:1347] Generated new cluster ID: 54db13b875ac45fd8e8224988d706b70
I20260927 00:47:16.550606 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:16.557119 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:16.558285 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:16.565824 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Generated new TSK 0
I20260927 00:47:16.566514 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:16.597554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:16.709904 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:16.710139 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:16.710179 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:16.714241 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:16.714294 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:16.714375 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:16.719460 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:16.720460 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:16.721490 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:16.731036 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.731043 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.731037 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.731386 15851 server_base.cc:1034] running on GCE node
I20260927 00:47:16.731778 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:16.732383 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:16.733559 15851 hybrid_clock.cc:648] HybridClock initialized: now 1790470036733542 us; error 36 us; skew 500 ppm
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:16.736431 15851 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:16.737287 15851 webserver.cc:460] Webserver started at http://127.0.98.1:42007/ using document root <none> and password file <none>
I20260927 00:47:16.737931 15851 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:16.738029 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:16.738246 15851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:16.740253 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c74cf829c944d9a8537d41f996685cd"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "1cd3a980012afc7928e46ce8b6a2b888"
server_key_iv: "d3e0ffef00a5abc2186670053e251374"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.740814 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c74cf829c944d9a8537d41f996685cd"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "1cd3a980012afc7928e46ce8b6a2b888"
server_key_iv: "d3e0ffef00a5abc2186670053e251374"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.744642 15851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:47:16.747081 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.748242 15851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:47:16.748384 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "4c74cf829c944d9a8537d41f996685cd"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "1cd3a980012afc7928e46ce8b6a2b888"
server_key_iv: "d3e0ffef00a5abc2186670053e251374"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.748498 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:16.772168 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:16.776645 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:16.776861 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:16.777634 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:16.778515 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:16.778592 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.778650 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:16.778683 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.788100 15851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 00:47:16.788146 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 00:47:16.789008 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:16.793991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15851
I20260927 00:47:16.794123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:16.794427 392 external_mini_cluster.cc:1428] Setting key 36f983aa2b00d65302ce46c29c8892a2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:16.811547 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.790815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53057 (local address 127.0.98.62:41447)
0927 00:47:16.790952 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.790956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.791906 (+ 950us) server_negotiation.cc:407] Connection header received
0927 00:47:16.792808 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.792812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.792877 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.792955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.794154 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.794817 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.796317 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.796542 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.796619 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.807412 (+ 10793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.807415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.807417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:16.807445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.809147 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:16.809676 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:16.809682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:16.809687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:16.809831 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:16.810250 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:16.810254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:16.810258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:16.810524 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.810640 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.811129 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.811282 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:47:16.812155 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.791121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:53057)
0927 00:47:16.791755 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.791796 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.792600 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.793120 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.793126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.793475 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.794004 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.794019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.794988 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.794992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.796182 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.796193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.796700 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.796707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.796938 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.803305 (+ 6367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:16.803331 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.807169 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.809293 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:16.809298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:16.809311 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:16.809567 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:16.809935 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:16.809938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:16.809939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:16.810145 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:16.810633 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.810639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.810880 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.811034 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.811298 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":127,"threads_started":1}
I20260927 00:47:16.813446 15980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:16.813728 15980 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:16.814358 15980 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260927 00:47:16.815915 15789 ts_manager.cc:194] Registered new tserver with Master: 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:16.817289 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:16.822890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:16.930863 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:16.931138 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:16.931213 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:16.935104 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:16.935187 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:16.935277 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:16.940583 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:16.941677 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:16.942781 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:16.951902 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.951898 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.951905 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.952090 15985 server_base.cc:1034] running on GCE node
I20260927 00:47:16.952715 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:16.953326 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:16.954506 15985 hybrid_clock.cc:648] HybridClock initialized: now 1790470036954497 us; error 34 us; skew 500 ppm
Sep 27 00:47:16 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:16.957247 15985 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:16.958231 15985 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260927 00:47:16.958777 15985 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:16.959043 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:16.959406 15985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:16.961421 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "bcd74760137b44af90842c9d08d30720"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "97bcd0dc5779270b7f270b529745c08d"
server_key_iv: "638b146d97472baf8e1dbb9d7c637593"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.962030 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd74760137b44af90842c9d08d30720"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "97bcd0dc5779270b7f270b529745c08d"
server_key_iv: "638b146d97472baf8e1dbb9d7c637593"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.965595 15985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:47:16.967996 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.969264 15985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:16.969401 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "bcd74760137b44af90842c9d08d30720"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "97bcd0dc5779270b7f270b529745c08d"
server_key_iv: "638b146d97472baf8e1dbb9d7c637593"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.969518 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:16.988021 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:16.991230 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:16.991451 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:16.992089 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:16.992950 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:16.993024 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.993101 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:16.993147 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.002491 15985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46831
I20260927 00:47:17.002507 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46831 every 8 connection(s)
I20260927 00:47:17.003419 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:17.008976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 15985
I20260927 00:47:17.009097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:17.009378 392 external_mini_cluster.cc:1428] Setting key bd96faf67d530d21550d2178bd6feaa7
Sep 27 00:47:17 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:17.018029 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.005249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45497 (local address 127.0.98.62:41447)
0927 00:47:17.005401 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.005405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.006513 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 00:47:17.007404 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.007408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.007463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.007544 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:17.008710 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.009547 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.010970 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.011192 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.011303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.014120 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:17.014123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:17.014125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:17.014148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:17.015704 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.016363 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.016366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.016367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.016445 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.016876 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.016879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.016880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.017050 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:17.017121 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.017747 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.017906 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:47:17.019313 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.005638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:45497)
0927 00:47:17.006361 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.006403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.007204 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.007707 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.007714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.008097 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:17.008568 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.008582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.009684 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.009688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.010855 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.010864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.011341 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.011346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.011511 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.012121 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:17.012137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:17.013949 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:17.015823 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:17.015832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:17.015851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:17.016244 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:17.016536 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:17.016541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:17.016546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:17.016772 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:17.017157 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:17.017166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:17.017487 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.017711 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.018095 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":188,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:47:17.020576 16114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:17.020831 16114 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:17.021327 16114 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:17.022486 15789 ts_manager.cc:194] Registered new tserver with Master: bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831)
I20260927 00:47:17.023111 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:17.037760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:17.147549 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:17.147841 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:17.147919 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:17.151981 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:17.152055 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:17.152140 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:17.157310 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:17.158334 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:17.159361 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:17.168895 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:17.168900 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.168963 16119 server_base.cc:1034] running on GCE node
W20260927 00:47:17.168895 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.169548 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:17.170183 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:17.171350 16119 hybrid_clock.cc:648] HybridClock initialized: now 1790470037171336 us; error 25 us; skew 500 ppm
Sep 27 00:47:17 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:17.174398 16119 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:17.175351 16119 webserver.cc:460] Webserver started at http://127.0.98.3:34145/ using document root <none> and password file <none>
I20260927 00:47:17.175951 16119 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:17.176046 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:17.176285 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:17.178496 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "15a019efca234d0dae0f6402d11dee28"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-l5sb"
server_key: "ea7a505d8d80e581594a4ce65fca033b"
server_key_iv: "32a4d29af7fd02ce243e14c6b0948f0b"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.179076 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15a019efca234d0dae0f6402d11dee28"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-l5sb"
server_key: "ea7a505d8d80e581594a4ce65fca033b"
server_key_iv: "32a4d29af7fd02ce243e14c6b0948f0b"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.183251 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:17.186029 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.187426 16119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:17.187557 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "15a019efca234d0dae0f6402d11dee28"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-l5sb"
server_key: "ea7a505d8d80e581594a4ce65fca033b"
server_key_iv: "32a4d29af7fd02ce243e14c6b0948f0b"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.187744 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:17.208884 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:17.211902 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:17.212107 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:17.212710 16119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:17.213526 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:17.213572 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.213639 16119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:17.213675 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.222189 16119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40095
I20260927 00:47:17.222216 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40095 every 8 connection(s)
I20260927 00:47:17.223073 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:17.224041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16119
I20260927 00:47:17.224160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:17.224462 392 external_mini_cluster.cc:1428] Setting key c0507a77a7aacfab736066cc75e02911
Sep 27 00:47:17 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:17.236819 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.225068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34485 (local address 127.0.98.62:41447)
0927 00:47:17.225200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.225204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.226314 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 00:47:17.227689 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.227692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.227742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.227819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:17.228946 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.229478 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.230666 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.230890 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.230998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.233419 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:17.233427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:17.233429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:17.233451 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:17.234830 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.235313 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.235318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.235320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.235402 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.235741 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.235746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.235748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.235927 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:17.236033 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.236514 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.236633 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:47:17.237694 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.225547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:34485)
0927 00:47:17.226186 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.226227 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.227488 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.227985 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.227991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.228341 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:17.228813 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.228826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.229598 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.229602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.230545 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.230554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.231030 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.231034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.231209 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.231837 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:17.231853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:17.233258 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:17.234942 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:17.234948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:17.234962 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:17.235210 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:17.235500 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:17.235503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:17.235505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:17.235641 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:17.236029 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:17.236035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:17.236285 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.236470 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.236803 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":221,"threads_started":1}
I20260927 00:47:17.238946 16248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:17.239190 16248 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:17.239671 16248 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260927 00:47:17.240649 15789 ts_manager.cc:194] Registered new tserver with Master: 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
I20260927 00:47:17.241276 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34485
I20260927 00:47:17.250679 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:17.260515 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.252711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.0.98.62:41447)
0927 00:47:17.252877 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.252881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.253101 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:47:17.253255 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.253264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.253311 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.253395 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:17.254201 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.254725 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.255597 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.255827 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.255932 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.257278 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:17.257282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:17.257285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:17.257318 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:17.258747 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.259220 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.259224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.259228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.259309 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:17.259595 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:17.259598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:17.259600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:17.259839 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:17.259984 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.260221 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.260343 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 00:47:17.263712 15789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:17.266812 15789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:17.282457 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.276611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.0.98.1:34745)
0927 00:47:17.277526 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.277546 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.277567 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:47:17.277643 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.277648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.277877 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.278040 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:17.278276 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279383 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.280922 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.281704 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.281885 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282036 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.282204 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.282297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":89,"threads_started":1}
I20260927 00:47:17.282447 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.276057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:38464)
0927 00:47:17.276888 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.276901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.276976 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.278028 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.278031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.278050 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:17.278154 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.278162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279536 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.279539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.280801 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.280810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.281888 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.281891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.281994 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282029 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.282172 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.282298 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":118,"thread_start_us":91,"threads_started":1}
I20260927 00:47:17.282871 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.276278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40095 (local address 127.0.0.1:40298)
0927 00:47:17.277177 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.277212 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.277298 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.278571 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.278574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.278591 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:17.278676 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.278682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279817 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.279820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.281102 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.281113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.282394 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.282398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.282571 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282593 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.282658 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.282720 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":103,"thread_start_us":103,"threads_started":1}
I20260927 00:47:17.283193 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.275877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46831 (local address 127.0.0.1:53600)
0927 00:47:17.276699 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.276760 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.276888 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.277737 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.277743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.277787 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:17.277938 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.277949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279468 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.279472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.281102 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.281113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.282394 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.282398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.282795 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282816 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.282891 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.283074 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":71,"thread_start_us":87,"threads_started":1}
I20260927 00:47:17.283371 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.276061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.98.2:46831)
0927 00:47:17.277217 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.277229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.277260 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:47:17.277341 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.277345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.277582 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.277755 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:17.278081 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279325 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.281248 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.282085 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.282199 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282327 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.283034 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.283134 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":135,"threads_started":1}
I20260927 00:47:17.283386 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.276729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.0.98.3:40095)
0927 00:47:17.278135 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.278146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.278167 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:47:17.278253 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.278267 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.278431 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.278580 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:17.278812 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.279654 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.281253 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.282085 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.282191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.282327 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.283129 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.283228 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":126,"threads_started":1}
I20260927 00:47:17.285876 16048 tablet_service.cc:1467] Processing CreateTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e (DEFAULT_TABLE table=test-table [id=924f862a36a74cd5bb8ebd32238b6df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:17.285907 16182 tablet_service.cc:1467] Processing CreateTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e (DEFAULT_TABLE table=test-table [id=924f862a36a74cd5bb8ebd32238b6df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:17.285908 15914 tablet_service.cc:1467] Processing CreateTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e (DEFAULT_TABLE table=test-table [id=924f862a36a74cd5bb8ebd32238b6df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:17.286854 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785ba7742ed04249a7f7e1ec2ba1045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:17.286903 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785ba7742ed04249a7f7e1ec2ba1045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:17.286912 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785ba7742ed04249a7f7e1ec2ba1045e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:17.293360 16265 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Bootstrap starting.
I20260927 00:47:17.294569 16266 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Bootstrap starting.
I20260927 00:47:17.296226 16266 tablet_bootstrap.cc:654] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:17.296263 16265 tablet_bootstrap.cc:654] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:17.296995 16265 log.cc:824] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:17.297271 16266 log.cc:824] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:17.297964 16267 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Bootstrap starting.
I20260927 00:47:17.299230 16266 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: No bootstrap required, opened a new log
I20260927 00:47:17.299230 16265 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: No bootstrap required, opened a new log
I20260927 00:47:17.299427 16266 ts_tablet_manager.cc:1397] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:47:17.299427 16265 ts_tablet_manager.cc:1397] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:47:17.299943 16267 tablet_bootstrap.cc:654] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:17.300973 16267 log.cc:824] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:17.302700 16267 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: No bootstrap required, opened a new log
I20260927 00:47:17.302901 16267 ts_tablet_manager.cc:1397] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:17.303321 16266 raft_consensus.cc:348] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.303321 16265 raft_consensus.cc:348] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.303602 16265 raft_consensus.cc:374] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:17.303602 16266 raft_consensus.cc:374] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:17.303664 16266 raft_consensus.cc:729] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a019efca234d0dae0f6402d11dee28, State: Initialized, Role: FOLLOWER
I20260927 00:47:17.303670 16265 raft_consensus.cc:729] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd74760137b44af90842c9d08d30720, State: Initialized, Role: FOLLOWER
I20260927 00:47:17.304128 16266 consensus_queue.cc:260] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.304128 16265 consensus_queue.cc:260] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.305127 16266 ts_tablet_manager.cc:1428] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:47:17.305127 16265 ts_tablet_manager.cc:1428] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:47:17.305449 16248 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
I20260927 00:47:17.305824 16114 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
I20260927 00:47:17.306589 16267 raft_consensus.cc:348] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.306846 16267 raft_consensus.cc:374] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:17.306898 16267 raft_consensus.cc:729] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cf829c944d9a8537d41f996685cd, State: Initialized, Role: FOLLOWER
I20260927 00:47:17.307461 16267 consensus_queue.cc:260] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.308159 15980 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
I20260927 00:47:17.308518 16267 ts_tablet_manager.cc:1428] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 00:47:17.425755 16272 raft_consensus.cc:482] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:17.426033 16272 raft_consensus.cc:504] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.427186 16272 leader_election.cc:288] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:17.431340 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.427655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60367 (local address 127.0.98.1:34745)
0927 00:47:17.427790 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.427800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.427814 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:17.427861 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.427863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.427902 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.427995 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:17.428310 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.428987 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.430214 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.430827 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.430903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.430955 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.431174 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.431228 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:47:17.431368 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.427483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.98.3:60367)
0927 00:47:17.427610 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.427627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.427717 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.428009 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.428013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.428029 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:17.428150 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.428155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.429113 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.429117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.430090 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.430099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.430939 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.430943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.431049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.431074 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.431151 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.431218 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:47:17.432060 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.427581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42395 (local address 127.0.98.2:46831)
0927 00:47:17.427773 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.427778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.428207 (+ 429us) server_negotiation.cc:407] Connection header received
0927 00:47:17.428320 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.428322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.428364 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.428421 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:17.428753 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.429397 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.430761 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.431369 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.431470 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.431537 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.431909 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.431972 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:47:17.432068 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.427483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46831 (local address 127.0.98.3:42395)
0927 00:47:17.428112 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.428123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.428210 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.428466 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.428469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.428487 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:17.428612 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.428618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.429532 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.429535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.430636 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.430647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.431620 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.431623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.431733 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.431782 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:47:17.431865 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:47:17.431930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":305,"thread_start_us":142,"threads_started":1}
I20260927 00:47:17.432090 15934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785ba7742ed04249a7f7e1ec2ba1045e" candidate_uuid: "15a019efca234d0dae0f6402d11dee28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cf829c944d9a8537d41f996685cd" is_pre_election: true
I20260927 00:47:17.432447 15934 raft_consensus.cc:2454] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a019efca234d0dae0f6402d11dee28 in term 0.
I20260927 00:47:17.432749 16068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785ba7742ed04249a7f7e1ec2ba1045e" candidate_uuid: "15a019efca234d0dae0f6402d11dee28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd74760137b44af90842c9d08d30720" is_pre_election: true
I20260927 00:47:17.432991 16136 leader_election.cc:302] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a019efca234d0dae0f6402d11dee28, 4c74cf829c944d9a8537d41f996685cd; no voters:
I20260927 00:47:17.433048 16068 raft_consensus.cc:2454] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a019efca234d0dae0f6402d11dee28 in term 0.
I20260927 00:47:17.433256 16272 raft_consensus.cc:2781] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:17.433362 16272 raft_consensus.cc:482] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:17.433418 16272 raft_consensus.cc:3037] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:17.434465 16272 raft_consensus.cc:504] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.434826 16272 leader_election.cc:288] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [CANDIDATE]: Term 1 election: Requested vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:17.435150 16068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785ba7742ed04249a7f7e1ec2ba1045e" candidate_uuid: "15a019efca234d0dae0f6402d11dee28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd74760137b44af90842c9d08d30720"
I20260927 00:47:17.435177 15934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785ba7742ed04249a7f7e1ec2ba1045e" candidate_uuid: "15a019efca234d0dae0f6402d11dee28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cf829c944d9a8537d41f996685cd"
I20260927 00:47:17.435277 15934 raft_consensus.cc:3037] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:17.435276 16068 raft_consensus.cc:3037] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:17.436507 16068 raft_consensus.cc:2454] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a019efca234d0dae0f6402d11dee28 in term 1.
I20260927 00:47:17.436507 15934 raft_consensus.cc:2454] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a019efca234d0dae0f6402d11dee28 in term 1.
I20260927 00:47:17.436805 16136 leader_election.cc:302] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a019efca234d0dae0f6402d11dee28, 4c74cf829c944d9a8537d41f996685cd; no voters:
I20260927 00:47:17.437000 16272 raft_consensus.cc:2781] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:17.437240 16272 raft_consensus.cc:686] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 1 LEADER]: Becoming Leader. State: Replica: 15a019efca234d0dae0f6402d11dee28, State: Running, Role: LEADER
I20260927 00:47:17.437575 16272 consensus_queue.cc:237] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:17.440681 15782 catalog_manager.cc:5495] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a019efca234d0dae0f6402d11dee28 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15a019efca234d0dae0f6402d11dee28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } health_report { overall_health: HEALTHY } } }
I20260927 00:47:17.470928 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.467485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.98.3:40095)
0927 00:47:17.467647 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.467652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.467676 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:47:17.467751 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.467754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.467816 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.467910 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:17.468135 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.468636 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.469500 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.469712 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.469887 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.470146 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:17.470585 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:17.470677 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.470764 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.470818 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 00:47:17.473965 16249 tablet.cc:2369] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:47:17.477303 15934 raft_consensus.cc:1261] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Refusing update from remote peer 15a019efca234d0dae0f6402d11dee28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:17.477315 16068 raft_consensus.cc:1261] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Refusing update from remote peer 15a019efca234d0dae0f6402d11dee28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:17.478108 16275 consensus_queue.cc:1035] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:17.478299 16272 consensus_queue.cc:1035] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:17.488456 16279 mvcc.cc:205] Tried to move back new op lower bound from 7333765273499521024 to 7333765273347219456. Current Snapshot: MvccSnapshot[applied={T|T < 7333765273499521024}]
I20260927 00:47:17.494783 15788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:17.494966 15788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:17.497385 15788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Sending DeleteTablet for 3 replicas of tablet 785ba7742ed04249a7f7e1ec2ba1045e
I20260927 00:47:17.498159 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53600
I20260927 00:47:17.498176 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38464
I20260927 00:47:17.498657 16288 tablet_replica.cc:330] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: stopping tablet replica
I20260927 00:47:17.498888 16182 tablet_service.cc:1514] Processing DeleteTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 00:47:17.499110 16289 tablet_replica.cc:330] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: stopping tablet replica
I20260927 00:47:17.499118 16288 raft_consensus.cc:2229] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:17.499344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15851
I20260927 00:47:17.499378 16289 raft_consensus.cc:2229] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:17.499648 16289 raft_consensus.cc:2258] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:17.500413 16288 raft_consensus.cc:2258] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:17.501646 16288 ts_tablet_manager.cc:1905] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:17.503866 16288 ts_tablet_manager.cc:1918] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:17.503974 16288 log.cc:1196] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/785ba7742ed04249a7f7e1ec2ba1045e
I20260927 00:47:17.504303 16288 ts_tablet_manager.cc:1939] T 785ba7742ed04249a7f7e1ec2ba1045e P bcd74760137b44af90842c9d08d30720: Deleting consensus metadata
I20260927 00:47:17.505430 15776 catalog_manager.cc:4841] TS bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): tablet 785ba7742ed04249a7f7e1ec2ba1045e (table test-table [id=924f862a36a74cd5bb8ebd32238b6df5]) successfully deleted
I20260927 00:47:17.505962 16290 tablet_replica.cc:330] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: stopping tablet replica
I20260927 00:47:17.506234 16290 raft_consensus.cc:2229] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:17.506597 16290 raft_consensus.cc:2258] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:17.507882 16290 ts_tablet_manager.cc:1905] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:47:17.508126 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34745 (error 108)
I20260927 00:47:17.509133 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.508824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:38466)
0927 00:47:17.508967 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.509045 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:47:17.509184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15985
I20260927 00:47:17.510375 16290 ts_tablet_manager.cc:1918] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:17.510480 16290 log.cc:1196] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/785ba7742ed04249a7f7e1ec2ba1045e
I20260927 00:47:17.510769 16290 ts_tablet_manager.cc:1939] T 785ba7742ed04249a7f7e1ec2ba1045e P 15a019efca234d0dae0f6402d11dee28: Deleting consensus metadata
I20260927 00:47:17.511759 15773 catalog_manager.cc:4841] TS 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): tablet 785ba7742ed04249a7f7e1ec2ba1045e (table test-table [id=924f862a36a74cd5bb8ebd32238b6df5]) successfully deleted
W20260927 00:47:17.512593 15774 catalog_manager.cc:4584] TS 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 785ba7742ed04249a7f7e1ec2ba1045e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
I20260927 00:47:17.518191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16119
I20260927 00:47:17.524083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 15757
I20260927 00:47:37.530848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--webserver_interface=127.0.98.62
--webserver_port=39303
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:37.641891 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:37.642140 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:37.642180 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:37.646242 16292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:47:37.646299 16292 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:47:37.646315 16292 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:47:37.646329 16292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:37.646347 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:37.646414 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:37.646445 16292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:37.646473 16292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:37.651875 16292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:37.652940 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:37.653982 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:37.666641 16301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:37.666708 16299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:37.666783 16292 server_base.cc:1034] running on GCE node
W20260927 00:47:37.666721 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:37.667527 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:37.668708 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:37.669855 16292 hybrid_clock.cc:648] HybridClock initialized: now 1790470057669809 us; error 57 us; skew 500 ppm
Sep 27 00:47:37 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:37.672766 16292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:37.673684 16292 webserver.cc:460] Webserver started at http://127.0.98.62:39303/ using document root <none> and password file <none>
I20260927 00:47:37.674281 16292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:37.674352 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:37.678232 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:37.680356 16308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:37.681617 16292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:37.681757 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "907ec10faa9d4b06800b79e44f2cb520"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "2ef3645f6b9e54749a725765e7f85e39"
server_key_iv: "7be138a1d65ca83af4c8c70583a060c2"
server_key_version: "encryptionkey@0"
I20260927 00:47:37.682358 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:37.706023 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:37.709369 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:37.709605 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:37.716930 16292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41447
I20260927 00:47:37.717221 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41447 every 8 connection(s)
I20260927 00:47:37.718173 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:37.722530 16361 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:47:37.724043 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap starting.
I20260927 00:47:37.728044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16292
I20260927 00:47:37.728485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34745
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=42007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:37.740861 16361 log.cc:824] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:37.749441 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:37.749910 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap complete.
I20260927 00:47:37.754220 16361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:37.754606 16361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Initialized, Role: FOLLOWER
I20260927 00:47:37.755079 16361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:37.755192 16361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:37.755285 16361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:37.755396 16361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:47:37.757220 16361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:37.757543 16361 leader_election.cc:302] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907ec10faa9d4b06800b79e44f2cb520; no voters:
I20260927 00:47:37.757892 16361 leader_election.cc:288] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:47:37.757974 16365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:47:37.758167 16365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 LEADER]: Becoming Leader. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Running, Role: LEADER
I20260927 00:47:37.758440 16365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:37.759068 16361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:37.759331 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907ec10faa9d4b06800b79e44f2cb520. Latest consensus state: current_term: 2 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:37.759449 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:37.759785 16369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:37.759822 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:37.760049 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:37.764410 16369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:37.765019 16369 catalog_manager.cc:1259] Loaded cluster ID: 54db13b875ac45fd8e8224988d706b70
I20260927 00:47:37.765084 16369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:37.767500 16369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:37.768226 16369 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Loaded TSK: 0
I20260927 00:47:37.784032 16369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Generated new TSK 1
I20260927 00:47:37.784267 16369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:47:37.842655 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:37.842971 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:37.843063 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:37.847151 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:37.847242 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:37.847342 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:37.852910 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42007
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:37.853960 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:37.854959 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:37.865007 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:37.865046 16388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:37.865003 16391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:37.865521 16363 server_base.cc:1034] running on GCE node
I20260927 00:47:37.865960 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:37.866708 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:37.867892 16363 hybrid_clock.cc:648] HybridClock initialized: now 1790470057867879 us; error 30 us; skew 500 ppm
Sep 27 00:47:37 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:37.876256 16363 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:37.877270 16363 webserver.cc:460] Webserver started at http://127.0.98.1:42007/ using document root <none> and password file <none>
I20260927 00:47:37.877913 16363 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:37.877990 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:37.881930 16363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:37.884215 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:37.885628 16363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:37.885808 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "4c74cf829c944d9a8537d41f996685cd"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "1cd3a980012afc7928e46ce8b6a2b888"
server_key_iv: "d3e0ffef00a5abc2186670053e251374"
server_key_version: "encryptionkey@0"
I20260927 00:47:37.886562 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:37.898425 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:37.902118 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:37.902324 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:37.902980 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:37.904044 16405 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:47:37.909138 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:47:37.909237 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:47:37.909343 16363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:47:37.911087 16363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:47:37.911154 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:47:37.911443 16405 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Bootstrap starting.
I20260927 00:47:37.922266 16363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 00:47:37.922261 16512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 00:47:37.923218 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:37.924866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16363
I20260927 00:47:37.925464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46831
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=46287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:37.929620 16405 log.cc:824] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:47:37 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:37.937331 16405 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:37.937835 16405 tablet_bootstrap.cc:492] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Bootstrap complete.
I20260927 00:47:37.939335 16405 ts_tablet_manager.cc:1397] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20260927 00:47:37.942320 16405 raft_consensus.cc:348] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:37.942762 16405 raft_consensus.cc:729] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cf829c944d9a8537d41f996685cd, State: Initialized, Role: FOLLOWER
I20260927 00:47:37.943233 16405 consensus_queue.cc:260] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } }
I20260927 00:47:37.944118 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.925453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:45107)
0927 00:47:37.926174 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:37.926242 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:37.927368 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:37.928768 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:37.928776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:37.929279 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:37.929934 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.929960 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.931657 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.931663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.933335 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.933349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.934388 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.934393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.934636 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.935474 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:37.935501 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:37.937704 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:37.940615 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:37.940621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:37.940634 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:37.940964 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:37.941367 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:37.941370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:37.941372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:37.941531 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:37.942281 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:37.942286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:37.942572 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 00:47:37.942754 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:47:37.943083 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":139,"threads_started":1}
I20260927 00:47:37.944146 16405 ts_tablet_manager.cc:1428] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:37.945387 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.926734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45107 (local address 127.0.98.62:41447)
0927 00:47:37.927270 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:37.927290 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:37.927339 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:47:37.927874 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:37.927890 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:37.928317 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:37.928673 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:37.930146 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.931481 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.933482 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.934228 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.934389 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.938066 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:37.938078 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:37.938097 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:37.938171 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:37.940476 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:37.941097 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:37.941110 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:37.941116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:37.941264 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:37.941641 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:37.941645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:37.941646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:37.942155 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:37.942289 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:37.944576 (+ 2287us) server_negotiation.cc:299] Negotiation successful
0927 00:47:37.944838 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":134,"threads_started":1}
I20260927 00:47:37.947427 16513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:37.947665 16513 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:37.948135 16513 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260927 00:47:37.950397 16325 ts_manager.cc:194] Registered new tserver with Master: 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:37.953910 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45107
I20260927 00:47:37.956342 16513 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
Sep 27 00:47:37 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:47:37.964298 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.953433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.98.1:34745)
0927 00:47:37.953849 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:37.953861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:37.954278 (+ 417us) server_negotiation.cc:407] Connection header received
0927 00:47:37.954462 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:37.954467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:37.954677 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:37.954832 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:37.955525 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.956546 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.957697 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.958084 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.958167 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.960673 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:37.960681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:37.960690 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:37.960710 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:37.962499 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:37.963023 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:37.963027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:37.963028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:37.963100 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:37.963371 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:37.963373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:37.963374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:37.963579 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:37.963708 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:37.964051 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 00:47:37.964176 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":98,"threads_started":1}
I20260927 00:47:37.964367 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.953348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:52874)
0927 00:47:37.954129 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:37.954178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:37.954319 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:37.954831 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:37.954837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:37.955263 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:37.955388 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.955395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.956660 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.956664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.957596 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.957606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.958179 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.958196 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.958304 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.958830 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:37.958857 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:37.960539 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:37.962609 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:37.962618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:37.962623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:37.962912 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:37.963185 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:37.963188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:37.963190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:37.963285 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:37.963719 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:37.963725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:37.963843 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 00:47:37.963987 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:47:37.964195 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":115,"threads_started":1}
I20260927 00:47:37.965291 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet 785ba7742ed04249a7f7e1ec2ba1045e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260927 00:47:37.966158 16522 tablet_replica.cc:330] stopping tablet replica
I20260927 00:47:37.966464 16522 raft_consensus.cc:2229] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:37.966717 16522 raft_consensus.cc:2258] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:37.967739 16522 ts_tablet_manager.cc:1905] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:37.972234 16522 ts_tablet_manager.cc:1918] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:37.972328 16522 log.cc:1196] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal/wals/785ba7742ed04249a7f7e1ec2ba1045e
I20260927 00:47:37.972685 16522 ts_tablet_manager.cc:1939] T 785ba7742ed04249a7f7e1ec2ba1045e P 4c74cf829c944d9a8537d41f996685cd: Deleting consensus metadata
I20260927 00:47:37.974550 16310 catalog_manager.cc:4841] TS 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745): tablet 785ba7742ed04249a7f7e1ec2ba1045e (table test-table [id=924f862a36a74cd5bb8ebd32238b6df5]) successfully deleted
W20260927 00:47:38.045022 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:38.045262 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:38.045303 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:38.049221 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:38.049274 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:38.049329 16516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:38.054409 16516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46287
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:38.055305 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:38.056213 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:38.065476 16530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:38.065565 16516 server_base.cc:1034] running on GCE node
W20260927 00:47:38.065474 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:38.065470 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:38.066159 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:38.066761 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:38.067929 16516 hybrid_clock.cc:648] HybridClock initialized: now 1790470058067911 us; error 30 us; skew 500 ppm
Sep 27 00:47:38 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:38.070663 16516 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:38.071650 16516 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260927 00:47:38.072203 16516 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:38.072276 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:38.076081 16516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:47:38.078162 16537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.079327 16516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:38.079464 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "bcd74760137b44af90842c9d08d30720"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "97bcd0dc5779270b7f270b529745c08d"
server_key_iv: "638b146d97472baf8e1dbb9d7c637593"
server_key_version: "encryptionkey@0"
I20260927 00:47:38.080029 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:38.091099 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:38.094128 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:38.094310 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:38.094872 16516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:38.095642 16516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:38.095711 16516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.095779 16516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:38.095829 16516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.104564 16516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46831
I20260927 00:47:38.104600 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46831 every 8 connection(s)
I20260927 00:47:38.105485 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:38.112474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16516
I20260927 00:47:38.112948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40095
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=34145
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:47:38 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:38.120265 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.107503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42111 (local address 127.0.98.62:41447)
0927 00:47:38.107637 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.107640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.108649 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 00:47:38.109517 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.109520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.109581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.109658 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.110782 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.111330 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.112629 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.112873 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.112964 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.116043 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:38.116072 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:38.116078 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:38.116116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:38.118069 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.118555 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.118561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.118563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.118663 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.118984 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.118988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.118991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.119263 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:38.119380 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.119855 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.120015 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:47:38.120872 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.107858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:42111)
0927 00:47:38.108488 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.108526 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.109316 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.109842 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.109848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.110183 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:38.110633 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.110647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.111456 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.111461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.112442 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.112454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.113035 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.113043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.113268 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.113999 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:38.114024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:38.115872 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:38.118185 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:38.118189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:38.118200 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:38.118454 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:38.118764 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:38.118766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:38.118767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:38.118890 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:38.119374 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:38.119379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:38.119617 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.119771 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.120018 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":131,"threads_started":1}
I20260927 00:47:38.122135 16651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:38.122433 16651 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:38.123070 16651 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260927 00:47:38.124204 16325 ts_manager.cc:194] Registered new tserver with Master: bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831)
I20260927 00:47:38.124773 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42111
W20260927 00:47:38.224534 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:38.224805 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:38.224848 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:38.228878 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:38.228932 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:38.229027 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:38.234378 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34145
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:38.235405 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:38.236485 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:38.245970 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:38.245965 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:38.246012 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:38.246243 16654 server_base.cc:1034] running on GCE node
I20260927 00:47:38.246726 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:38.247355 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:38.248556 16654 hybrid_clock.cc:648] HybridClock initialized: now 1790470058248524 us; error 46 us; skew 500 ppm
Sep 27 00:47:38 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:38.251358 16654 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:38.252377 16654 webserver.cc:460] Webserver started at http://127.0.98.3:34145/ using document root <none> and password file <none>
I20260927 00:47:38.252928 16654 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:38.253024 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:38.256855 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:38.258993 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.260320 16654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:47:38.260409 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "15a019efca234d0dae0f6402d11dee28"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-l5sb"
server_key: "ea7a505d8d80e581594a4ce65fca033b"
server_key_iv: "32a4d29af7fd02ce243e14c6b0948f0b"
server_key_version: "encryptionkey@0"
I20260927 00:47:38.261004 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:38.281580 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:38.284755 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:38.284893 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:38.285562 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:38.286361 16654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:38.286414 16654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.286453 16654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:38.286473 16654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:38.295372 16654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40095
I20260927 00:47:38.295392 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40095 every 8 connection(s)
I20260927 00:47:38.296311 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:38.299890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16654
Sep 27 00:47:38 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:38.309597 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.300668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.98.62:41447)
0927 00:47:38.300894 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.300898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.301020 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:47:38.301154 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.301157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.301193 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.301261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.302202 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.302968 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.304151 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.304374 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.304452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.306111 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:38.306115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:38.306118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:38.306141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:38.307759 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.308232 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.308238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.308241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.308336 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.308693 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.308699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.308703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.308932 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:38.309156 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.309317 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.309413 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":62,"threads_started":1}
I20260927 00:47:38.311502 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.298194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53395 (local address 127.0.98.62:41447)
0927 00:47:38.298327 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.298331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.299606 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 00:47:38.300706 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.300710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.300767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.300850 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.302491 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.303140 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.304587 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.304786 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.305002 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.307426 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:38.307430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:38.307433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:38.307473 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:38.309131 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.309813 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.309816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.309818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.309899 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.310312 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.310315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.310316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.310537 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:38.310629 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.311203 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.311332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:47:38.312328 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.298595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:53395)
0927 00:47:38.299438 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.299480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.300469 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.301145 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.301154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.301678 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:38.302366 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.302382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.303253 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.303256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.304466 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.304475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.305038 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.305042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.305210 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.305751 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:38.305812 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:38.307274 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:38.309242 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:38.309248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:38.309263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:38.309664 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:38.309994 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:38.309997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:38.309998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:38.310207 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:38.310637 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:38.310641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:38.310936 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.311159 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.311490 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":104,"threads_started":1}
I20260927 00:47:38.313340 16783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260927 00:47:38.313603 16783 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:38.314111 16783 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260927 00:47:38.315026 16325 ts_manager.cc:194] Registered new tserver with Master: 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
I20260927 00:47:38.315562 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53395
I20260927 00:47:38.323282 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:38.327611 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.323969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.0.98.62:41447)
0927 00:47:38.324141 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.324147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.324315 (+ 168us) server_negotiation.cc:407] Connection header received
0927 00:47:38.324503 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.324513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.324620 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.324736 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:38.324979 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.325494 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.326465 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.326682 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.326792 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.327091 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:38.327217 (+ 126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:47:38.327476 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58882: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:47:38.327665 16517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58882: token expired
W20260927 00:47:38.327955 16252 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41447)
I20260927 00:47:38.335228 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.328398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.98.62:41447)
0927 00:47:38.328506 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.328511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.328547 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:47:38.328722 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.328725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.328768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.328847 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.329441 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.329963 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.330757 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.330960 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.331072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.332336 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:38.332340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:38.332343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:38.332373 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:38.333599 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.334083 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.334086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.334088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.334166 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:38.334429 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:38.334434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:38.334436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:38.334605 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:38.334741 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.334958 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.335075 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:47:38.337584 16325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:38.340853 16325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:38.348599 16447 tablet_service.cc:1467] Processing CreateTablet for tablet 780cc14cbc8640d68967952d08c601e6 (DEFAULT_TABLE table=test-table [id=277ba886246344abb2c2cbadef554089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:38.349272 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780cc14cbc8640d68967952d08c601e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:38.352124 16405 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Bootstrap starting.
I20260927 00:47:38.352950 16405 tablet_bootstrap.cc:654] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:38.353741 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.347620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46831 (local address 127.0.0.1:56378)
0927 00:47:38.347778 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.347794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.347869 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.348886 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.348889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.348909 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:38.349013 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.349018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.350537 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.350544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.351807 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.351818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.353206 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.353211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.353342 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.353405 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.353512 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.353596 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 00:47:38.353808 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.347895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.0.98.2:46831)
0927 00:47:38.348326 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.348337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.348361 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:47:38.348445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.348452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.348696 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.349359 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:38.349552 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.350425 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.352075 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.352994 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.353186 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.353358 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.353545 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.353639 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":106,"threads_started":1}
I20260927 00:47:38.354135 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.348327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.3:40095)
0927 00:47:38.348700 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.348711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.348734 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:47:38.348809 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.348814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.349014 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.349660 (+ 646us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:38.349881 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.350927 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.352370 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.353147 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.353687 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.353850 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.353902 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.353996 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":154,"threads_started":1}
I20260927 00:47:38.354455 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.347875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40095 (local address 127.0.0.1:37370)
0927 00:47:38.348153 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.348167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.348250 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.349176 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.349179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.349196 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:38.349306 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.349316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.351038 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.351042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.352234 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.352245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.353271 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.353274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.353384 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.353422 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.353504 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.353581 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":11,"thread_start_us":123,"threads_started":1}
I20260927 00:47:38.354892 16405 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: No bootstrap required, opened a new log
I20260927 00:47:38.355000 16405 ts_tablet_manager.cc:1397] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:47:38.355471 16405 raft_consensus.cc:348] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.355589 16405 raft_consensus.cc:374] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:38.355628 16405 raft_consensus.cc:729] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cf829c944d9a8537d41f996685cd, State: Initialized, Role: FOLLOWER
I20260927 00:47:38.355742 16405 consensus_queue.cc:260] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.355984 16405 ts_tablet_manager.cc:1428] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:47:38.356841 16585 tablet_service.cc:1467] Processing CreateTablet for tablet 780cc14cbc8640d68967952d08c601e6 (DEFAULT_TABLE table=test-table [id=277ba886246344abb2c2cbadef554089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:38.357066 16717 tablet_service.cc:1467] Processing CreateTablet for tablet 780cc14cbc8640d68967952d08c601e6 (DEFAULT_TABLE table=test-table [id=277ba886246344abb2c2cbadef554089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:38.357857 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780cc14cbc8640d68967952d08c601e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:38.358069 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780cc14cbc8640d68967952d08c601e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:38.364694 16797 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Bootstrap starting.
I20260927 00:47:38.365309 16796 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Bootstrap starting.
I20260927 00:47:38.367061 16797 tablet_bootstrap.cc:654] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:38.367344 16796 tablet_bootstrap.cc:654] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:38.367808 16797 log.cc:824] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:38.368028 16796 log.cc:824] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:38.369530 16797 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: No bootstrap required, opened a new log
I20260927 00:47:38.369607 16796 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: No bootstrap required, opened a new log
I20260927 00:47:38.369809 16797 ts_tablet_manager.cc:1397] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:38.369848 16796 ts_tablet_manager.cc:1397] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:38.372802 16796 raft_consensus.cc:348] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.372802 16797 raft_consensus.cc:348] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.373013 16796 raft_consensus.cc:374] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:38.373041 16797 raft_consensus.cc:374] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:38.373073 16796 raft_consensus.cc:729] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a019efca234d0dae0f6402d11dee28, State: Initialized, Role: FOLLOWER
I20260927 00:47:38.373111 16797 raft_consensus.cc:729] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd74760137b44af90842c9d08d30720, State: Initialized, Role: FOLLOWER
I20260927 00:47:38.373500 16796 consensus_queue.cc:260] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.373502 16797 consensus_queue.cc:260] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.374135 16651 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
I20260927 00:47:38.374135 16783 heartbeater.cc:503] Master 127.0.98.62:41447 was elected leader, sending a full tablet report...
I20260927 00:47:38.374505 16796 ts_tablet_manager.cc:1428] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:38.374513 16797 ts_tablet_manager.cc:1428] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260927 00:47:38.424400 16514 tablet.cc:2369] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:47:38.440092 16801 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:38.440308 16801 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.441633 16801 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095), 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:38.446944 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.442055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40095 (local address 127.0.98.2:35229)
0927 00:47:38.442394 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.442410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.442509 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.442996 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.442999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.443014 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:38.443192 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.443201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.444229 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.444232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.445435 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.445444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.446455 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.446459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.446617 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.446643 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.446729 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.446809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":4}
I20260927 00:47:38.446954 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.442263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35229 (local address 127.0.98.3:40095)
0927 00:47:38.442676 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.442682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.442696 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:38.442750 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.442755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.442807 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.442877 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:38.443332 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.444096 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.445553 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.446304 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.446427 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.446499 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.446743 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.446829 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260927 00:47:38.447338 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.442592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48683 (local address 127.0.98.1:34745)
0927 00:47:38.442727 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.442732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.442759 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:47:38.442813 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.442816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.442881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.442980 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:38.443647 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.444496 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.446032 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.446750 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.446847 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.447005 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.447140 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.447220 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:47:38.447678 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.442105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.98.2:48683)
0927 00:47:38.442453 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.442468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.442570 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.443089 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.443092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.443108 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:38.443483 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.443494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.444677 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.444680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.445874 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.445884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.446861 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.446864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.446974 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.447001 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:47:38.447492 (+ 491us) client_negotiation.cc:240] Negotiation successful
0927 00:47:38.447567 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":29,"thread_start_us":145,"threads_started":1}
I20260927 00:47:38.447611 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780cc14cbc8640d68967952d08c601e6" candidate_uuid: "bcd74760137b44af90842c9d08d30720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a019efca234d0dae0f6402d11dee28" is_pre_election: true
I20260927 00:47:38.447937 16737 raft_consensus.cc:2454] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd74760137b44af90842c9d08d30720 in term 0.
I20260927 00:47:38.448216 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780cc14cbc8640d68967952d08c601e6" candidate_uuid: "bcd74760137b44af90842c9d08d30720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cf829c944d9a8537d41f996685cd" is_pre_election: true
I20260927 00:47:38.448463 16538 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720; no voters:
I20260927 00:47:38.448489 16466 raft_consensus.cc:2454] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd74760137b44af90842c9d08d30720 in term 0.
I20260927 00:47:38.448745 16801 raft_consensus.cc:2781] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:38.448834 16801 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:38.448899 16801 raft_consensus.cc:3037] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:38.449990 16801 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.450371 16801 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [CANDIDATE]: Term 1 election: Requested vote from peers 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095), 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745)
I20260927 00:47:38.450672 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780cc14cbc8640d68967952d08c601e6" candidate_uuid: "bcd74760137b44af90842c9d08d30720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a019efca234d0dae0f6402d11dee28"
I20260927 00:47:38.450735 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780cc14cbc8640d68967952d08c601e6" candidate_uuid: "bcd74760137b44af90842c9d08d30720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cf829c944d9a8537d41f996685cd"
I20260927 00:47:38.450809 16737 raft_consensus.cc:3037] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:38.450852 16466 raft_consensus.cc:3037] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:38.452054 16737 raft_consensus.cc:2454] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd74760137b44af90842c9d08d30720 in term 1.
I20260927 00:47:38.452054 16466 raft_consensus.cc:2454] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd74760137b44af90842c9d08d30720 in term 1.
I20260927 00:47:38.452349 16538 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720; no voters:
I20260927 00:47:38.452502 16801 raft_consensus.cc:2781] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:38.452718 16801 raft_consensus.cc:686] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 1 LEADER]: Becoming Leader. State: Replica: bcd74760137b44af90842c9d08d30720, State: Running, Role: LEADER
I20260927 00:47:38.453068 16801 consensus_queue.cc:237] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:38.456017 16325 catalog_manager.cc:5495] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd74760137b44af90842c9d08d30720 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcd74760137b44af90842c9d08d30720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:38.477851 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.474570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.98.2:46831)
0927 00:47:38.474706 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.474709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.474722 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:47:38.474859 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.474861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.474915 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.475007 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:38.475216 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.475723 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.476514 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.476729 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.476818 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.477059 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:38.477463 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:38.477558 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:38.477613 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 00:47:38.477662 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:38.483579 16737 raft_consensus.cc:1261] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Refusing update from remote peer bcd74760137b44af90842c9d08d30720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:38.483626 16466 raft_consensus.cc:1261] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Refusing update from remote peer bcd74760137b44af90842c9d08d30720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:38.484290 16801 consensus_queue.cc:1035] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:38.484511 16803 consensus_queue.cc:1035] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:38.489934 16806 mvcc.cc:205] Tried to move back new op lower bound from 7333765359541891072 to 7333765359426199552. Current Snapshot: MvccSnapshot[applied={T|T < 7333765359541891072}]
I20260927 00:47:38.490132 16807 mvcc.cc:205] Tried to move back new op lower bound from 7333765359541891072 to 7333765359426199552. Current Snapshot: MvccSnapshot[applied={T|T < 7333765359541891072}]
I20260927 00:47:38.493885 16808 mvcc.cc:205] Tried to move back new op lower bound from 7333765359541891072 to 7333765359426199552. Current Snapshot: MvccSnapshot[applied={T|T < 7333765359541891072}]
I20260927 00:47:38.499150 16323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:38.499289 16323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:38.501573 16323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Sending DeleteTablet for 3 replicas of tablet 780cc14cbc8640d68967952d08c601e6
I20260927 00:47:38.502302 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet 780cc14cbc8640d68967952d08c601e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56378
I20260927 00:47:38.502416 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet 780cc14cbc8640d68967952d08c601e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260927 00:47:38.502497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16363
I20260927 00:47:38.502790 16717 tablet_service.cc:1514] Processing DeleteTablet for tablet 780cc14cbc8640d68967952d08c601e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37370
I20260927 00:47:38.503208 16817 tablet_replica.cc:330] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: stopping tablet replica
I20260927 00:47:38.503509 16817 raft_consensus.cc:2229] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:38.503770 16815 tablet_replica.cc:330] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: stopping tablet replica
I20260927 00:47:38.503805 16817 raft_consensus.cc:2258] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:38.504041 16815 raft_consensus.cc:2229] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:38.504403 16815 raft_consensus.cc:2258] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:38.505263 16817 ts_tablet_manager.cc:1905] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:38.505708 16815 ts_tablet_manager.cc:1905] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:38.508169 16817 ts_tablet_manager.cc:1918] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:38.508312 16815 ts_tablet_manager.cc:1918] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:38.508342 16817 log.cc:1196] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/780cc14cbc8640d68967952d08c601e6
I20260927 00:47:38.508401 16815 log.cc:1196] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/780cc14cbc8640d68967952d08c601e6
I20260927 00:47:38.508777 16815 ts_tablet_manager.cc:1939] T 780cc14cbc8640d68967952d08c601e6 P bcd74760137b44af90842c9d08d30720: Deleting consensus metadata
I20260927 00:47:38.508793 16817 ts_tablet_manager.cc:1939] T 780cc14cbc8640d68967952d08c601e6 P 15a019efca234d0dae0f6402d11dee28: Deleting consensus metadata
W20260927 00:47:38.508755 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34745 (error 108)
I20260927 00:47:38.509856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16516
I20260927 00:47:38.509884 16309 catalog_manager.cc:4841] TS 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): tablet 780cc14cbc8640d68967952d08c601e6 (table test-table [id=277ba886246344abb2c2cbadef554089]) successfully deleted
I20260927 00:47:38.509884 16312 catalog_manager.cc:4841] TS bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): tablet 780cc14cbc8640d68967952d08c601e6 (table test-table [id=277ba886246344abb2c2cbadef554089]) successfully deleted
I20260927 00:47:38.511812 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.511470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:52882)
0927 00:47:38.511634 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.511712 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:47:38.511986 16310 catalog_manager.cc:4584] TS 4c74cf829c944d9a8537d41f996685cd (127.0.98.1:34745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 780cc14cbc8640d68967952d08c601e6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
I20260927 00:47:38.516893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16654
I20260927 00:47:38.522424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16292
I20260927 00:47:58.528813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--webserver_interface=127.0.98.62
--webserver_port=39303
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:58.637094 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:58.637351 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:58.637390 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:58.641362 16819 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:47:58.641412 16819 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:47:58.641427 16819 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:47:58.641440 16819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:58.641458 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:58.641516 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:58.641546 16819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:58.641572 16819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:58.646759 16819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:58.647822 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:58.648854 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:58.657619 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:58.657701 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:58.659893 16819 server_base.cc:1034] running on GCE node
W20260927 00:47:58.659941 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:58.660494 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:58.661588 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:58.662801 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790470078662802 us; error 51 us; skew 500 ppm
Sep 27 00:47:58 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:58.669348 16819 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:58.670344 16819 webserver.cc:460] Webserver started at http://127.0.98.62:39303/ using document root <none> and password file <none>
I20260927 00:47:58.670926 16819 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:58.671020 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:58.674897 16819 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:47:58.676966 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:58.678241 16819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:47:58.678385 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "907ec10faa9d4b06800b79e44f2cb520"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "2ef3645f6b9e54749a725765e7f85e39"
server_key_iv: "7be138a1d65ca83af4c8c70583a060c2"
server_key_version: "encryptionkey@0"
I20260927 00:47:58.678944 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:58.693987 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:58.697186 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:58.697386 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:58.704365 16819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41447
I20260927 00:47:58.704635 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41447 every 8 connection(s)
I20260927 00:47:58.705454 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:58.709625 16887 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:47:58.711234 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap starting.
I20260927 00:47:58.715149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16819
I20260927 00:47:58.715576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34745
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=42007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:58.732602 16887 log.cc:824] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:58.746793 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:58.747256 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: Bootstrap complete.
I20260927 00:47:58.752674 16887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:58.753197 16887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Initialized, Role: FOLLOWER
I20260927 00:47:58.753726 16887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:58.753919 16887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:58.753993 16887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:58.754091 16887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:47:58.756445 16887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:58.756819 16887 leader_election.cc:302] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907ec10faa9d4b06800b79e44f2cb520; no voters:
I20260927 00:47:58.757169 16887 leader_election.cc:288] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:47:58.757368 16891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:47:58.757632 16891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [term 3 LEADER]: Becoming Leader. State: Replica: 907ec10faa9d4b06800b79e44f2cb520, State: Running, Role: LEADER
I20260927 00:47:58.757956 16891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260927 00:47:58.758518 16887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:58.759464 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907ec10faa9d4b06800b79e44f2cb520. Latest consensus state: current_term: 3 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:58.759604 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:58.759478 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "907ec10faa9d4b06800b79e44f2cb520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907ec10faa9d4b06800b79e44f2cb520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260927 00:47:58.759706 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:58.763795 16906 catalog_manager.cc:1259] Loaded cluster ID: 54db13b875ac45fd8e8224988d706b70
I20260927 00:47:58.763901 16906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: loading cluster ID for follower catalog manager: success
I20260927 00:47:58.766430 16906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 907ec10faa9d4b06800b79e44f2cb520: acquiring CA information for follower catalog manager: success
F20260927 00:47:58.767711 16906 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f61a94049ad google::LogMessage::Fail() at ??:0
@ 0x7f61a940788a google::LogMessage::SendToLog() at ??:0
@ 0x7f61a94044de google::LogMessage::Flush() at ??:0
@ 0x7f61a9405a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f61ac0c818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f61b3d6d989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f61b3d4a023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f61b3d4a54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f61b3d43652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f61b3d429e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f61b3d89541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f61ab647512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61acc8b6db start_thread at ??:0
@ 0x7f61a7f0261f clone at ??:0
W20260927 00:47:58.865729 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:58.866047 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:58.866091 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:58.870121 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:58.870170 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:58.870255 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:58.875404 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42007
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:58.876439 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:58.877458 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:58.887218 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:58.887218 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:58.887542 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:58.887845 16889 server_base.cc:1034] running on GCE node
I20260927 00:47:58.888345 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:58.889148 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:58.890436 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790470078890398 us; error 47 us; skew 500 ppm
Sep 27 00:47:58 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:58.894028 16889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:58.895160 16889 webserver.cc:460] Webserver started at http://127.0.98.1:42007/ using document root <none> and password file <none>
I20260927 00:47:58.895882 16889 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:58.895974 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:58.900988 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:58.904064 16937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:58.905597 16889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:58.905719 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "4c74cf829c944d9a8537d41f996685cd"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "1cd3a980012afc7928e46ce8b6a2b888"
server_key_iv: "d3e0ffef00a5abc2186670053e251374"
server_key_version: "encryptionkey@0"
I20260927 00:47:58.906461 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:58.932027 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:58.935806 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:58.935992 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:58.936717 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:58.938126 16944 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:47:58.942139 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:47:58.942242 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 00:47:58.942359 16889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:47:58.944823 16889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:47:58.944877 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260927 00:47:58.945196 16944 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Bootstrap starting.
I20260927 00:47:58.955832 16889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 00:47:58.956171 17051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 00:47:58.957038 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:58.963373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 16889
I20260927 00:47:58.963949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46831
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=46287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:58.965202 16944 log.cc:824] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:58.971773 16944 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:58.972138 16944 tablet_bootstrap.cc:492] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Bootstrap complete.
I20260927 00:47:58.973205 16944 ts_tablet_manager.cc:1397] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.000s
I20260927 00:47:58.975626 16944 raft_consensus.cc:348] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:58.976037 16944 raft_consensus.cc:729] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cf829c944d9a8537d41f996685cd, State: Initialized, Role: FOLLOWER
I20260927 00:47:58.976514 16944 consensus_queue.cc:260] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:47:58.977401 16944 ts_tablet_manager.cc:1428] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 00:47:59.002535 17054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:58.959413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:44363)
0927 00:47:58.960035 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:58.960107 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:58.961168 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:59.001571 (+ 40403us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":140,"threads_started":1}
W20260927 00:47:59.003144 17052 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41447: Client connection negotiation failed: client connection to 127.0.98.62:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:47:59.003959 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.003673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:47193)
0927 00:47:59.003821 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.003897 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:47:59.004732 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.004518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:53531)
0927 00:47:59.004628 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.004677 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:47:59.004932 17052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:47:59.087673 17055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:59.087913 17055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:59.087990 17055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:59.092001 17055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:59.092051 17055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:59.092144 17055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:59.097227 17055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46287
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:59.098512 17055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:59.099727 17055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:59.109229 17062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:59.109217 17065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:59.109275 17063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:59.109570 17055 server_base.cc:1034] running on GCE node
I20260927 00:47:59.110034 17055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:59.110656 17055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:59.111809 17055 hybrid_clock.cc:648] HybridClock initialized: now 1790470079111783 us; error 36 us; skew 500 ppm
Sep 27 00:47:59 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:59.114513 17055 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:59.115415 17055 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260927 00:47:59.115962 17055 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:59.116020 17055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:59.119709 17055 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:47:59.121634 17072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.122826 17055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:59.122967 17055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "bcd74760137b44af90842c9d08d30720"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-l5sb"
server_key: "97bcd0dc5779270b7f270b529745c08d"
server_key_iv: "638b146d97472baf8e1dbb9d7c637593"
server_key_version: "encryptionkey@0"
I20260927 00:47:59.123522 17055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:59.135727 17055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:59.138649 17055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:59.138849 17055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:59.139420 17055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:59.140221 17055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:59.140292 17055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.140362 17055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:59.140410 17055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.149670 17055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46831
I20260927 00:47:59.149703 17185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46831 every 8 connection(s)
I20260927 00:47:59.150692 17055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:59.154560 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.153042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:33649)
0927 00:47:59.153694 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.153853 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":117,"threads_started":1}
W20260927 00:47:59.155033 17186 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41447: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
I20260927 00:47:59.155681 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.155491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:59177)
0927 00:47:59.155587 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.155632 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:47:59.156404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17055
I20260927 00:47:59.156459 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.156234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:54787)
0927 00:47:59.156352 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.156401 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:47:59.156646 17186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:47:59.156863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40095
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41447
--webserver_port=34145
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:59.265602 17189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:59.265978 17189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:59.266072 17189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:59.270174 17189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:59.270274 17189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:59.270372 17189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:59.275775 17189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34145
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:47:59.276928 17189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:59.278033 17189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:59.288189 17194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:59.288192 17195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:59.288169 17197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:59.288264 17189 server_base.cc:1034] running on GCE node
I20260927 00:47:59.288782 17189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:59.289484 17189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:59.290681 17189 hybrid_clock.cc:648] HybridClock initialized: now 1790470079290634 us; error 59 us; skew 500 ppm
Sep 27 00:47:59 dist-test-slave-l5sb krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:59.293553 17189 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:59.294420 17189 webserver.cc:460] Webserver started at http://127.0.98.3:34145/ using document root <none> and password file <none>
I20260927 00:47:59.294921 17189 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:59.294981 17189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:59.298707 17189 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:47:59.300665 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.302019 17189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:47:59.302150 17189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "15a019efca234d0dae0f6402d11dee28"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-l5sb"
server_key: "ea7a505d8d80e581594a4ce65fca033b"
server_key_iv: "32a4d29af7fd02ce243e14c6b0948f0b"
server_key_version: "encryptionkey@0"
I20260927 00:47:59.302695 17189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:59.328088 17189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:59.331072 17189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:59.331266 17189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:59.331851 17189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:59.332679 17189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:59.332748 17189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.332819 17189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:59.332871 17189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:59.342067 17189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40095
I20260927 00:47:59.342083 17317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40095 every 8 connection(s)
I20260927 00:47:59.343093 17189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:59.347111 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.345314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:48957)
0927 00:47:59.346271 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.346381 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":621,"threads_started":1}
W20260927 00:47:59.347560 17318 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41447: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
I20260927 00:47:59.348134 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.347940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:37799)
0927 00:47:59.348037 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.348083 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 00:47:59.348795 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:59.348623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:40099)
0927 00:47:59.348714 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:59.348751 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:47:59.349009 17318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:47:59.353045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17189
I20260927 00:48:00.006062 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.005644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:35801)
0927 00:48:00.005898 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:00.005953 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260927 00:48:00.157543 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.157169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:39893)
0927 00:48:00.157395 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:00.157444 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260927 00:48:00.343422 17324 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:00.343664 17324 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:48:00.344956 17324 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
I20260927 00:48:00.350138 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.349665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:57023)
0927 00:48:00.349973 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:00.350025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
Sep 27 00:48:00 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:48:00 dist-test-slave-l5sb krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:48:00.360188 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.345264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40095 (local address 127.0.98.1:52629)
0927 00:48:00.345891 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:00.345903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:00.345988 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:00.348335 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:00.348338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:00.348509 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:00.348763 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:00.348776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.350623 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.350627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:00.351563 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:00.351573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.352469 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.352472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:00.352562 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:00.353244 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:00.353262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:00.354966 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:00.357851 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:00.357854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:00.357856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:00.358098 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:00.358593 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:00.358596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:00.358698 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:00.359334 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:00.359337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:00.359486 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 00:48:00.359625 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 00:48:00.359950 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":162,"thread_start_us":88,"threads_started":1}
I20260927 00:48:00.360235 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.345617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42307 (local address 127.0.98.2:46831)
0927 00:48:00.346072 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:00.346097 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:00.346133 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:00.346979 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:00.346986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:00.347456 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:00.347811 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:00.348949 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.350383 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.351687 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.352215 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.352376 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:00.355530 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:00.355539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:00.355555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:00.355591 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:00.357483 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358120 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:00.358129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:00.358137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:00.358280 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358692 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:00.358695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:00.358696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:00.359123 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:00.359222 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:00.359796 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 00:48:00.360067 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":102,"threads_started":1}
I20260927 00:48:00.360287 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.345401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52629 (local address 127.0.98.3:40095)
0927 00:48:00.345872 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:00.345888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:00.345996 (+ 108us) server_negotiation.cc:407] Connection header received
0927 00:48:00.346974 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:00.346983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:00.347456 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:00.347811 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:00.348947 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.350481 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.351693 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.352345 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.352518 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:00.355177 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:00.355182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:00.355194 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:00.355218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:00.357483 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358231 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:00.358236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:00.358240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:00.358484 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358810 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:00.358813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:00.358814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:00.359218 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:00.359364 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:00.359804 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 00:48:00.360096 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":105,"threads_started":1}
I20260927 00:48:00.361626 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:00.345214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46831 (local address 127.0.98.1:42307)
0927 00:48:00.345594 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:00.345609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:00.345705 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:00.347866 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:00.347871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:00.348248 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:00.348762 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:00.348776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.350538 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.350548 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:00.351560 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:00.351569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:00.352362 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:00.352367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:00.352553 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:00.353213 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:00.353245 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:00.355338 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:00.357601 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:00.357606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:00.357616 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:00.358010 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:00.358402 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:00.358406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:00.358408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:00.358596 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:00.359248 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:00.359258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:00.359507 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 00:48:00.359663 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 00:48:00.360058 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":138,"spinlock_wait_cycles":27264}
W20260927 00:48:00.362823 16938 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
W20260927 00:48:00.363621 16941 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
I20260927 00:48:00.363740 16941 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cf829c944d9a8537d41f996685cd; no voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720
I20260927 00:48:00.364074 17324 raft_consensus.cc:2726] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:48:01.007328 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:01.006840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:36695)
0927 00:48:01.007110 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:01.007193 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":115,"threads_started":1}
I20260927 00:48:01.158727 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:01.158349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:55235)
0927 00:48:01.158590 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:01.158641 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260927 00:48:01.351271 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:01.350862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:54839)
0927 00:48:01.351126 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:01.351184 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260927 00:48:02.008724 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:02.008201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:42927)
0927 00:48:02.008496 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:02.008597 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
I20260927 00:48:02.160064 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:02.159615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:39415)
0927 00:48:02.159902 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:02.159967 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260927 00:48:02.245610 17334 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:02.245782 17334 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:48:02.246296 17334 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
W20260927 00:48:02.247056 16941 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
W20260927 00:48:02.247169 16938 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
I20260927 00:48:02.247254 16941 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cf829c944d9a8537d41f996685cd; no voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720
I20260927 00:48:02.247445 17334 raft_consensus.cc:2726] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:48:02.352625 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:02.352198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:49093)
0927 00:48:02.352455 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:02.352514 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260927 00:48:03.010111 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:03.009659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:50725)
0927 00:48:03.009920 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:03.009987 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260927 00:48:03.161245 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:03.160822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:33575)
0927 00:48:03.161088 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:03.161140 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260927 00:48:03.353731 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:03.353310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:54995)
0927 00:48:03.353576 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:03.353634 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260927 00:48:04.011482 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:04.011054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:36013)
0927 00:48:04.011327 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:04.011386 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260927 00:48:04.162453 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:04.162055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:45635)
0927 00:48:04.162308 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:04.162362 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260927 00:48:04.355033 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:04.354575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:46245)
0927 00:48:04.354879 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:04.354937 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
I20260927 00:48:05.012681 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:05.012204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:54789)
0927 00:48:05.012523 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:05.012583 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260927 00:48:05.087352 17343 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:05.087508 17343 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:48:05.087965 17343 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
W20260927 00:48:05.088708 16941 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
W20260927 00:48:05.088886 16938 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
I20260927 00:48:05.088980 16938 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cf829c944d9a8537d41f996685cd; no voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720
I20260927 00:48:05.089139 17343 raft_consensus.cc:2726] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:48:05.163612 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:05.163153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:54621)
0927 00:48:05.163456 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:05.163515 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260927 00:48:05.356266 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:05.355742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:49879)
0927 00:48:05.356033 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:05.356098 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260927 00:48:06.013928 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:06.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:59413)
0927 00:48:06.013707 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:06.013819 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260927 00:48:06.164880 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:06.164438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:50269)
0927 00:48:06.164718 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:06.164771 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":70,"threads_started":1}
I20260927 00:48:06.357491 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:06.357040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:38993)
0927 00:48:06.357348 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:06.357401 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":72,"threads_started":1}
I20260927 00:48:07.000861 17349 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:07.000998 17349 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:48:07.001487 17349 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
W20260927 00:48:07.002321 16941 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
W20260927 00:48:07.002441 16938 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
I20260927 00:48:07.002557 16938 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cf829c944d9a8537d41f996685cd; no voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720
I20260927 00:48:07.002724 17349 raft_consensus.cc:2726] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:48:07.015122 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:07.014662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:53113)
0927 00:48:07.014980 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:07.015028 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260927 00:48:07.166102 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:07.165638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:36649)
0927 00:48:07.165952 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:07.166010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":72,"threads_started":1}
I20260927 00:48:07.358736 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:07.358300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:42401)
0927 00:48:07.358559 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:07.358621 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
I20260927 00:48:08.016448 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:08.015999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:39669)
0927 00:48:08.016278 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:08.016341 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260927 00:48:08.167366 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:08.166948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:55851)
0927 00:48:08.167204 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:08.167266 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260927 00:48:08.360042 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:08.359555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:50483)
0927 00:48:08.359850 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:08.359919 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260927 00:48:09.018035 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:09.017525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:57671)
0927 00:48:09.017849 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:09.017924 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260927 00:48:09.168601 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:09.168137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:49495)
0927 00:48:09.168435 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:09.168498 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260927 00:48:09.361325 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:09.360929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:48161)
0927 00:48:09.361172 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:09.361227 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260927 00:48:10.019450 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:10.019014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:41119)
0927 00:48:10.019294 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:10.019352 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260927 00:48:10.169893 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:10.169461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:43817)
0927 00:48:10.169691 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:10.169744 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260927 00:48:10.362457 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:10.362065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:52113)
0927 00:48:10.362309 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:10.362360 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260927 00:48:11.020740 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:11.020286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:50657)
0927 00:48:11.020555 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:11.020610 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260927 00:48:11.171172 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:11.170717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:40883)
0927 00:48:11.171004 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:11.171066 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260927 00:48:11.363730 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:11.363319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:51963)
0927 00:48:11.363576 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:11.363636 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260927 00:48:11.886579 17365 raft_consensus.cc:482] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:11.886754 17365 raft_consensus.cc:504] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd74760137b44af90842c9d08d30720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46831 } } peers { permanent_uuid: "15a019efca234d0dae0f6402d11dee28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40095 } } peers { permanent_uuid: "4c74cf829c944d9a8537d41f996685cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 00:48:11.887319 17365 leader_election.cc:288] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831), 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095)
W20260927 00:48:11.888064 16938 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 15a019efca234d0dae0f6402d11dee28 (127.0.98.3:40095): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
W20260927 00:48:11.888163 16941 leader_election.cc:341] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcd74760137b44af90842c9d08d30720 (127.0.98.2:46831): Not found: Tablet not found: 780cc14cbc8640d68967952d08c601e6
I20260927 00:48:11.888218 16941 leader_election.cc:302] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cf829c944d9a8537d41f996685cd; no voters: 15a019efca234d0dae0f6402d11dee28, bcd74760137b44af90842c9d08d30720
I20260927 00:48:11.888365 17365 raft_consensus.cc:2726] T 780cc14cbc8640d68967952d08c601e6 P 4c74cf829c944d9a8537d41f996685cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:48:12.021999 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:12.021489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:45665)
0927 00:48:12.021826 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:12.021902 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260927 00:48:12.172369 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:12.171918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:56631)
0927 00:48:12.172195 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:12.172251 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260927 00:48:12.364945 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:12.364555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:57731)
0927 00:48:12.364791 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:12.364852 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260927 00:48:13.023074 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:13.022685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:45337)
0927 00:48:13.022927 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:13.022980 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260927 00:48:13.173666 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:13.173256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:57879)
0927 00:48:13.173512 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:13.173570 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260927 00:48:13.366159 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:13.365701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:52185)
0927 00:48:13.366011 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:13.366065 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":160,"threads_started":1}
I20260927 00:48:14.024398 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:14.023989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:49237)
0927 00:48:14.024249 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:14.024304 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260927 00:48:14.174964 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:14.174536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:41081)
0927 00:48:14.174808 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:14.174867 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260927 00:48:14.353753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 16889
I20260927 00:48:14.360057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17055
I20260927 00:48:14.365340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17189
2026-09-27T00:48:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59187 ms)
[----------] 30 tests from SecurityITest (239868 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:14 dist-test-slave-l5sb krb5kdc[17376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:14 dist-test-slave-l5sb krb5kdc[17376](info): set up 2 sockets
Sep 27 00:48:14 dist-test-slave-l5sb krb5kdc[17376](info): commencing operation
krb5kdc: starting...
W20260927 00:48:15.406327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:15 dist-test-slave-l5sb krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:15.535315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:15.647229 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:15.647549 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:15.647629 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:15.651751 17389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:15.651829 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:15.651866 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:15.651921 17389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:15.651961 17389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:15.657150 17389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:15.658247 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:15.659237 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:15.667781 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:15.667816 17389 server_base.cc:1034] running on GCE node
W20260927 00:48:15.668615 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:15.668828 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:15.669334 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:15.670511 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:15.671720 17389 hybrid_clock.cc:648] HybridClock initialized: now 1790470095671684 us; error 46 us; skew 500 ppm
Sep 27 00:48:15 dist-test-slave-l5sb krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:15.674507 17389 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:15.675396 17389 webserver.cc:460] Webserver started at http://127.0.98.62:35867/ using document root <none> and password file <none>
I20260927 00:48:15.675940 17389 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:15.676025 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:15.676256 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:15.678282 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "ce6115251cad4649a6fa6ac7046f5a8c"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "9dfda5934243ae228782215e566dabcd"
server_key_iv: "a3b8140f42b34789e9e271d4b24a5d3b"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.678825 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce6115251cad4649a6fa6ac7046f5a8c"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "9dfda5934243ae228782215e566dabcd"
server_key_iv: "a3b8140f42b34789e9e271d4b24a5d3b"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.682583 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:48:15.685066 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:15.686467 17389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:15.686630 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "ce6115251cad4649a6fa6ac7046f5a8c"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "9dfda5934243ae228782215e566dabcd"
server_key_iv: "a3b8140f42b34789e9e271d4b24a5d3b"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.686756 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:15.705500 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:15.708554 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:15.708772 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:15.716351 17389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38759
I20260927 00:48:15.716379 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38759 every 8 connection(s)
I20260927 00:48:15.717366 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:15.720114 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:15.721669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17389
I20260927 00:48:15.721823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:15.722153 392 external_mini_cluster.cc:1428] Setting key b7d78fb968698408ada80b747c4781e7
I20260927 00:48:15.726195 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: Bootstrap starting.
I20260927 00:48:15.728312 17457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:15.729014 17457 log.cc:824] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:48:15 dist-test-slave-l5sb krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:15.731178 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: No bootstrap required, opened a new log
I20260927 00:48:15.734759 17457 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } }
I20260927 00:48:15.734995 17457 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:15.735085 17457 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6115251cad4649a6fa6ac7046f5a8c, State: Initialized, Role: FOLLOWER
I20260927 00:48:15.735549 17457 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } }
I20260927 00:48:15.735718 17457 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:15.735791 17457 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:15.735901 17457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:15.736263 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:15.723615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.98.62:38759)
0927 00:48:15.724043 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:15.724058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:15.724094 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:15.724708 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:15.724724 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:15.725034 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:15.725375 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:15.726238 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:15.727264 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:15.728116 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:15.728649 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:15.728815 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:15.731683 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:15.731688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:15.731701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:15.731727 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:15.733346 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:15.733817 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:15.733822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:15.733826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:15.733932 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:15.734230 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:15.734233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:15.734235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:15.734618 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:15.734839 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:15.735075 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 00:48:15.735401 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260927 00:48:15.736976 17457 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } }
I20260927 00:48:15.737321 17457 leader_election.cc:302] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce6115251cad4649a6fa6ac7046f5a8c; no voters:
I20260927 00:48:15.737684 17457 leader_election.cc:288] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:15.737749 17462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:15.737975 17462 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [term 1 LEADER]: Becoming Leader. State: Replica: ce6115251cad4649a6fa6ac7046f5a8c, State: Running, Role: LEADER
I20260927 00:48:15.738319 17462 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } }
I20260927 00:48:15.738955 17457 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:15.740141 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } } }
I20260927 00:48:15.740141 17464 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce6115251cad4649a6fa6ac7046f5a8c. Latest consensus state: current_term: 1 leader_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6115251cad4649a6fa6ac7046f5a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38759 } } }
I20260927 00:48:15.740351 17464 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:15.740351 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:15.740763 17472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:15.743655 17472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:15.749238 17472 catalog_manager.cc:1347] Generated new cluster ID: cc1e6d0d37c4482a90aec80fd5460563
I20260927 00:48:15.749321 17472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:15.773175 17472 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:15.774152 17472 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:15.778633 17472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: Generated new TSK 0
I20260927 00:48:15.779501 17472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:48:15.818883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:15.954393 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:15.954641 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:15.954725 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:15.958743 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:15.958799 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:15.958886 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:15.964193 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:15.965188 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:15.966223 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:15.975441 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:15.975430 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:15.975430 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:15.975841 17483 server_base.cc:1034] running on GCE node
I20260927 00:48:15.976244 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:15.976861 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:15.978037 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790470095978023 us; error 35 us; skew 500 ppm
Sep 27 00:48:15 dist-test-slave-l5sb krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:15.980818 17483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:48:15.981808 17483 webserver.cc:460] Webserver started at http://127.0.98.1:35947/ using document root <none> and password file <none>
I20260927 00:48:15.982367 17483 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:15.982455 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:15.982674 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:15.984654 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8f0a903a0c649b59505c32474cb09d7"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "0e5f71ebf2aae01ee5c118242d085d6d"
server_key_iv: "9a513d03d0dd1b636f1fe248f425dc98"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.985242 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8f0a903a0c649b59505c32474cb09d7"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "0e5f71ebf2aae01ee5c118242d085d6d"
server_key_iv: "9a513d03d0dd1b636f1fe248f425dc98"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.988747 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:15.991227 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:15.992350 17483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:15.992472 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "c8f0a903a0c649b59505c32474cb09d7"
format_stamp: "Formatted at 2026-09-27 00:48:15 on dist-test-slave-l5sb"
server_key: "0e5f71ebf2aae01ee5c118242d085d6d"
server_key_iv: "9a513d03d0dd1b636f1fe248f425dc98"
server_key_version: "encryptionkey@0"
I20260927 00:48:15.992569 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:16.009882 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:16.012589 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:16.012777 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:16.013325 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:16.014171 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:16.014235 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.014299 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:16.014380 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.023656 17483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44949
I20260927 00:48:16.023679 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44949 every 8 connection(s)
I20260927 00:48:16.024518 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:16.025498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17483
I20260927 00:48:16.025635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:16.025988 392 external_mini_cluster.cc:1428] Setting key 24755bc1d880ca34cfeb320e07227747
Sep 27 00:48:16 dist-test-slave-l5sb krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:48:16.039515 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.026425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37631 (local address 127.0.98.62:38759)
0927 00:48:16.026582 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.026587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.027577 (+ 990us) server_negotiation.cc:407] Connection header received
0927 00:48:16.029559 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.029564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.029626 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.029726 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:16.030933 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.031896 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.033021 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.033468 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.033588 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.035977 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:16.035980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:16.035982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:16.036005 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:16.037398 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.037919 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.037922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.037924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.037999 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.038366 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.038372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.038377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.038582 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:16.038691 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.039217 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.039338 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:48:16.040330 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.026894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38759 (local address 127.0.98.1:37631)
0927 00:48:16.027422 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.027480 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.029287 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.029922 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.029929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.030309 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:16.030786 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.030799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.032022 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.032025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.032899 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.032908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.033613 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.033616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.033753 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.034378 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:16.034394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:16.035823 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:16.037495 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.037502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.037520 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.037835 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.038093 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.038098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.038100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.038268 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.038692 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:16.038699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:16.038940 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.039122 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.039446 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260927 00:48:16.041549 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38759
I20260927 00:48:16.041832 17612 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:16.042429 17612 heartbeater.cc:511] Master 127.0.98.62:38759 requested a full tablet report, sending...
I20260927 00:48:16.044031 17421 ts_manager.cc:194] Registered new tserver with Master: c8f0a903a0c649b59505c32474cb09d7 (127.0.98.1:44949)
I20260927 00:48:16.045142 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:48:16.055188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:16.164412 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:16.164649 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:16.164690 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:16.168540 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:16.168587 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:16.168641 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:16.173653 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:16.174590 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:16.175526 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:16.184959 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:16.185082 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:16.184916 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:16.185245 17617 server_base.cc:1034] running on GCE node
I20260927 00:48:16.185695 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:16.186424 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:16.187618 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790470096187602 us; error 26 us; skew 500 ppm
Sep 27 00:48:16 dist-test-slave-l5sb krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:16.190536 17617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:48:16.191665 17617 webserver.cc:460] Webserver started at http://127.0.98.2:39095/ using document root <none> and password file <none>
I20260927 00:48:16.192262 17617 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:16.192355 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:16.192584 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:16.194816 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "0393818808c64e8ab7bbb710b8300d2e"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "85283c9d5d2e7262e548cda71462d992"
server_key_iv: "015e88714a92a960c6951ea3add98f1b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.195401 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0393818808c64e8ab7bbb710b8300d2e"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "85283c9d5d2e7262e548cda71462d992"
server_key_iv: "015e88714a92a960c6951ea3add98f1b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.199343 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:48:16.201972 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.203138 17617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:16.203269 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "0393818808c64e8ab7bbb710b8300d2e"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "85283c9d5d2e7262e548cda71462d992"
server_key_iv: "015e88714a92a960c6951ea3add98f1b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.203373 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:16.222963 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:16.225932 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:16.226119 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:16.226679 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:16.227464 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:16.227532 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.227610 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:16.227659 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.236711 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44217
I20260927 00:48:16.236729 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44217 every 8 connection(s)
I20260927 00:48:16.237608 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:16.241494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17617
I20260927 00:48:16.241623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:16.241932 392 external_mini_cluster.cc:1428] Setting key af0216b777045848cf62e78d3e48f3b8
Sep 27 00:48:16 dist-test-slave-l5sb krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:48:16.253263 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.239396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41239 (local address 127.0.98.62:38759)
0927 00:48:16.239563 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.239568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.240580 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 00:48:16.242421 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.242425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.242498 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.242592 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:16.244076 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.244640 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.245961 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.246282 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.246371 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.249433 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:16.249436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:16.249438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:16.249463 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:16.251098 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.251695 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.251700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.251702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.251810 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.252157 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.252161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.252163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.252351 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:16.252456 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.252964 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.253090 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:48:16.254140 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.239923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38759 (local address 127.0.98.2:41239)
0927 00:48:16.240421 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.240464 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.242171 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.242789 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.242797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.243222 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:16.243919 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.243941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.244768 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.244774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.245844 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.245855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.246455 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.246462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.246722 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.247378 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:16.247404 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:16.249098 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:16.251277 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.251284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.251300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.251578 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.251907 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.251910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.251912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.252059 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.252455 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:16.252462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:16.252719 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.252922 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.253254 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:48:16.255349 17746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38759
I20260927 00:48:16.255579 17746 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:16.256091 17746 heartbeater.cc:511] Master 127.0.98.62:38759 requested a full tablet report, sending...
I20260927 00:48:16.257169 17421 ts_manager.cc:194] Registered new tserver with Master: 0393818808c64e8ab7bbb710b8300d2e (127.0.98.2:44217)
I20260927 00:48:16.257742 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:48:16.270308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:16.387353 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:16.387612 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:16.387660 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:16.392084 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:16.392141 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:16.392197 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:16.397538 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:16.398559 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:16.399528 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:16.409389 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:16.409413 17751 server_base.cc:1034] running on GCE node
W20260927 00:48:16.409608 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:16.409397 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:16.410132 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:16.410810 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:16.411994 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790470096411956 us; error 53 us; skew 500 ppm
Sep 27 00:48:16 dist-test-slave-l5sb krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:16.414819 17751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:48:16.415695 17751 webserver.cc:460] Webserver started at http://127.0.98.3:41337/ using document root <none> and password file <none>
I20260927 00:48:16.416230 17751 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:16.416290 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:16.416452 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:16.418466 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "cea90775b4b248fab113fe562c530672"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "a8f80b7d7b8c87880d59344ed7b92c53"
server_key_iv: "353d9ee3b1215c7145877e655f66d95b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.418967 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cea90775b4b248fab113fe562c530672"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "a8f80b7d7b8c87880d59344ed7b92c53"
server_key_iv: "353d9ee3b1215c7145877e655f66d95b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.422612 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:48:16.424906 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.426199 17751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:16.426293 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "cea90775b4b248fab113fe562c530672"
format_stamp: "Formatted at 2026-09-27 00:48:16 on dist-test-slave-l5sb"
server_key: "a8f80b7d7b8c87880d59344ed7b92c53"
server_key_iv: "353d9ee3b1215c7145877e655f66d95b"
server_key_version: "encryptionkey@0"
I20260927 00:48:16.426421 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:16.456137 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:16.458928 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:16.459141 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:16.459724 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:16.460507 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:16.460553 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.460621 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:16.460659 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:16.469502 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38459
I20260927 00:48:16.469522 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38459 every 8 connection(s)
I20260927 00:48:16.470392 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:16.476460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17751
I20260927 00:48:16.476591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:16.476931 392 external_mini_cluster.cc:1428] Setting key 82d2215751a6ada227731e64fd930679
Sep 27 00:48:16 dist-test-slave-l5sb krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:16.484223 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.472125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33657 (local address 127.0.98.62:38759)
0927 00:48:16.472267 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.472271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.473097 (+ 826us) server_negotiation.cc:407] Connection header received
0927 00:48:16.474326 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.474329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.474376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.474450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:16.475561 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.476123 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.477675 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.478010 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.478125 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.480677 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:16.480680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:16.480682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:16.480708 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:16.482265 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.482763 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.482766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.482767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.482850 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.483179 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.483182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.483183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.483369 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:16.483458 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.483946 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.484053 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:16.484972 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.472542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38759 (local address 127.0.98.3:33657)
0927 00:48:16.472951 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.472992 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.474131 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.474600 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.474607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.474946 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:16.475431 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.475445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.476278 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.476283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.477539 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.477549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.478169 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.478174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.478395 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.479020 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:16.479044 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:16.480517 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:16.482384 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.482389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.482401 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.482653 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.482931 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:16.482933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:16.482934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:16.483065 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:16.483456 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:16.483461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:16.483735 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.483894 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.484143 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260927 00:48:16.486099 17880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38759
I20260927 00:48:16.486354 17880 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:16.486865 17880 heartbeater.cc:511] Master 127.0.98.62:38759 requested a full tablet report, sending...
I20260927 00:48:16.487861 17421 ts_manager.cc:194] Registered new tserver with Master: cea90775b4b248fab113fe562c530672 (127.0.98.3:38459)
I20260927 00:48:16.488502 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33657
I20260927 00:48:16.490808 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:16.499857 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.492768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.62:38759)
0927 00:48:16.492881 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.492886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.493041 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:48:16.493126 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.493128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.493162 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.493254 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:16.493970 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.494481 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.495275 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.495474 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.495548 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.496926 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:16.496929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:16.496932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:16.496955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:16.498261 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.498662 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.498667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.498669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.498754 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:16.499016 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:16.499020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:16.499021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:16.499238 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:16.499373 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.499600 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.499708 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:48:16.502288 17421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:16.505131 17421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:16.519477 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.513249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38459 (local address 127.0.0.1:34630)
0927 00:48:16.513553 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.513596 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.513720 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.515019 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.515030 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.515057 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:16.515262 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.515271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516839 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.516842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.518008 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.518018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.519084 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.519087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.519183 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.519202 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.519280 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.519334 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":67,"thread_start_us":97,"threads_started":1}
I20260927 00:48:16.519688 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.514075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.1:44949)
0927 00:48:16.514699 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.514713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.514732 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:48:16.514794 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.514798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.514976 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.515131 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:16.515405 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516629 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.518123 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.518681 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.518801 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.518945 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.519430 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.519528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":94,"threads_started":1}
I20260927 00:48:16.519356 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.513509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44949 (local address 127.0.0.1:58920)
0927 00:48:16.514153 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.514170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.514258 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.515107 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.515109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.515130 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:16.515284 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.515291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516767 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.516770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.518030 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.518041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.518803 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.518807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.518910 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.518945 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.519073 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.519189 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":55,"thread_start_us":94,"threads_started":1}
I20260927 00:48:16.519852 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.514010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44217 (local address 127.0.0.1:38756)
0927 00:48:16.514300 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.514314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.514406 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.515388 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.515390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.515405 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:16.515471 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.515475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516437 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.516440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.517957 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.517971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.519241 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.519244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.519335 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.519358 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.519428 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.519496 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":113,"thread_start_us":88,"threads_started":1}
I20260927 00:48:16.520268 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.514245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.0.98.2:44217)
0927 00:48:16.514676 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.514689 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.514710 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:48:16.514788 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.514796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.515075 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.515234 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:16.515570 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516304 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.518100 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.518921 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.519041 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.519179 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.520006 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.520098 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":111,"threads_started":1}
I20260927 00:48:16.519749 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.513424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.0.98.3:38459)
0927 00:48:16.514499 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.514512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.514534 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:48:16.514625 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.514631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.514837 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.515416 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:16.515585 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.516676 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.518161 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.518921 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.519096 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.519207 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.519311 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.519412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":184,"threads_started":1}
I20260927 00:48:16.522851 17814 tablet_service.cc:1467] Processing CreateTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 (DEFAULT_TABLE table=test-table [id=f820866b6688433c8ffa58c2438f87e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:16.522853 17680 tablet_service.cc:1467] Processing CreateTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 (DEFAULT_TABLE table=test-table [id=f820866b6688433c8ffa58c2438f87e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:16.522853 17546 tablet_service.cc:1467] Processing CreateTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 (DEFAULT_TABLE table=test-table [id=f820866b6688433c8ffa58c2438f87e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:16.523799 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda8e7ffa494bf3bbc09c298f4f1498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:16.523836 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda8e7ffa494bf3bbc09c298f4f1498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:16.523837 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebda8e7ffa494bf3bbc09c298f4f1498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:16.529808 17897 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Bootstrap starting.
I20260927 00:48:16.531399 17898 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Bootstrap starting.
I20260927 00:48:16.532267 17897 tablet_bootstrap.cc:654] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:16.533098 17898 tablet_bootstrap.cc:654] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:16.533349 17897 log.cc:824] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:16.534093 17899 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Bootstrap starting.
I20260927 00:48:16.534291 17898 log.cc:824] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:16.535612 17897 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: No bootstrap required, opened a new log
I20260927 00:48:16.535805 17897 ts_tablet_manager.cc:1397] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:16.535977 17899 tablet_bootstrap.cc:654] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:16.536919 17899 log.cc:824] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:16.538781 17899 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: No bootstrap required, opened a new log
I20260927 00:48:16.538961 17899 ts_tablet_manager.cc:1397] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:48:16.539422 17898 tablet_bootstrap.cc:492] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: No bootstrap required, opened a new log
I20260927 00:48:16.539361 17897 raft_consensus.cc:348] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.539675 17898 ts_tablet_manager.cc:1397] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:48:16.539685 17897 raft_consensus.cc:374] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:16.539742 17897 raft_consensus.cc:729] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea90775b4b248fab113fe562c530672, State: Initialized, Role: FOLLOWER
I20260927 00:48:16.540247 17897 consensus_queue.cc:260] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.540926 17880 heartbeater.cc:503] Master 127.0.98.62:38759 was elected leader, sending a full tablet report...
I20260927 00:48:16.541378 17897 ts_tablet_manager.cc:1428] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:48:16.542593 17899 raft_consensus.cc:348] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.542809 17899 raft_consensus.cc:374] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:16.542868 17899 raft_consensus.cc:729] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f0a903a0c649b59505c32474cb09d7, State: Initialized, Role: FOLLOWER
I20260927 00:48:16.543277 17899 consensus_queue.cc:260] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.543345 17898 raft_consensus.cc:348] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.543637 17898 raft_consensus.cc:374] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:16.543697 17898 raft_consensus.cc:729] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0393818808c64e8ab7bbb710b8300d2e, State: Initialized, Role: FOLLOWER
I20260927 00:48:16.544095 17898 consensus_queue.cc:260] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.544206 17612 heartbeater.cc:503] Master 127.0.98.62:38759 was elected leader, sending a full tablet report...
I20260927 00:48:16.544212 17899 ts_tablet_manager.cc:1428] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:48:16.544973 17898 ts_tablet_manager.cc:1428] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:48:16.545373 17746 heartbeater.cc:503] Master 127.0.98.62:38759 was elected leader, sending a full tablet report...
W20260927 00:48:16.721302 17881 tablet.cc:2369] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:16.738755 17747 tablet.cc:2369] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:16.775846 17613 tablet.cc:2369] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:16.815977 17905 raft_consensus.cc:482] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:16.816221 17905 raft_consensus.cc:504] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.817353 17905 leader_election.cc:288] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea90775b4b248fab113fe562c530672 (127.0.98.3:38459), c8f0a903a0c649b59505c32474cb09d7 (127.0.98.1:44949)
I20260927 00:48:16.821167 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.817693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55843 (local address 127.0.98.3:38459)
0927 00:48:16.817852 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.817857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.818033 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:48:16.818184 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.818218 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.818262 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.818344 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:16.818637 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.819139 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820248 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.820635 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.820807 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.820975 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.821045 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:48:16.821184 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.817546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38459 (local address 127.0.98.2:55843)
0927 00:48:16.817931 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.817948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.818069 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.818356 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.818359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.818376 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:16.818528 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.818534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.819249 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.819255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.820109 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.820117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.820748 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.820847 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.820874 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.820957 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.821034 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":42,"thread_start_us":127,"threads_started":1}
I20260927 00:48:16.821192 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.817915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58319 (local address 127.0.98.1:44949)
0927 00:48:16.818070 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.818077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.818094 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:48:16.818301 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.818304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.818344 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.818424 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:16.818728 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.819214 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820259 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.820647 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820759 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.820819 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.821007 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.821073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:48:16.821214 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.817698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44949 (local address 127.0.98.2:58319)
0927 00:48:16.818012 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.818027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:16.818180 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:16.818435 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:16.818438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:16.818454 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:16.818615 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.818621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.819319 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.819325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.820144 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:16.820152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.820761 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.820765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:16.820872 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.820896 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:48:16.820980 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:48:16.821059 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":33,"thread_start_us":84,"threads_started":1}
I20260927 00:48:16.821887 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebda8e7ffa494bf3bbc09c298f4f1498" candidate_uuid: "0393818808c64e8ab7bbb710b8300d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea90775b4b248fab113fe562c530672" is_pre_election: true
I20260927 00:48:16.822048 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebda8e7ffa494bf3bbc09c298f4f1498" candidate_uuid: "0393818808c64e8ab7bbb710b8300d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f0a903a0c649b59505c32474cb09d7" is_pre_election: true
I20260927 00:48:16.822280 17834 raft_consensus.cc:2454] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0393818808c64e8ab7bbb710b8300d2e in term 0.
I20260927 00:48:16.822324 17566 raft_consensus.cc:2454] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0393818808c64e8ab7bbb710b8300d2e in term 0.
I20260927 00:48:16.822805 17633 leader_election.cc:302] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0393818808c64e8ab7bbb710b8300d2e, cea90775b4b248fab113fe562c530672; no voters:
I20260927 00:48:16.823064 17905 raft_consensus.cc:2781] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:16.823159 17905 raft_consensus.cc:482] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:16.823215 17905 raft_consensus.cc:3037] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:16.824241 17905 raft_consensus.cc:504] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.824584 17905 leader_election.cc:288] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [CANDIDATE]: Term 1 election: Requested vote from peers cea90775b4b248fab113fe562c530672 (127.0.98.3:38459), c8f0a903a0c649b59505c32474cb09d7 (127.0.98.1:44949)
I20260927 00:48:16.824934 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebda8e7ffa494bf3bbc09c298f4f1498" candidate_uuid: "0393818808c64e8ab7bbb710b8300d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea90775b4b248fab113fe562c530672"
I20260927 00:48:16.825016 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebda8e7ffa494bf3bbc09c298f4f1498" candidate_uuid: "0393818808c64e8ab7bbb710b8300d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f0a903a0c649b59505c32474cb09d7"
I20260927 00:48:16.825059 17834 raft_consensus.cc:3037] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:16.825138 17566 raft_consensus.cc:3037] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:16.826201 17834 raft_consensus.cc:2454] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0393818808c64e8ab7bbb710b8300d2e in term 1.
I20260927 00:48:16.826207 17566 raft_consensus.cc:2454] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0393818808c64e8ab7bbb710b8300d2e in term 1.
I20260927 00:48:16.826479 17633 leader_election.cc:302] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0393818808c64e8ab7bbb710b8300d2e, cea90775b4b248fab113fe562c530672; no voters:
I20260927 00:48:16.826642 17905 raft_consensus.cc:2781] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:16.826892 17905 raft_consensus.cc:686] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 1 LEADER]: Becoming Leader. State: Replica: 0393818808c64e8ab7bbb710b8300d2e, State: Running, Role: LEADER
I20260927 00:48:16.827205 17905 consensus_queue.cc:237] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } }
I20260927 00:48:16.830197 17421 catalog_manager.cc:5495] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0393818808c64e8ab7bbb710b8300d2e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0393818808c64e8ab7bbb710b8300d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0393818808c64e8ab7bbb710b8300d2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 } health_report { overall_health: UNKNOWN } } }
I20260927 00:48:16.881771 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.878489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.0.98.2:44217)
0927 00:48:16.878619 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:16.878622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:16.878634 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:48:16.878712 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:16.878714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:16.878767 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:16.878854 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:16.879058 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.879569 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.880396 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:16.880606 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:16.880687 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:16.880920 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:16.881345 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:16.881435 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:16.881553 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 00:48:16.881603 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:48:16.887488 17834 raft_consensus.cc:1261] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 1 FOLLOWER]: Refusing update from remote peer 0393818808c64e8ab7bbb710b8300d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:16.887488 17566 raft_consensus.cc:1261] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 1 FOLLOWER]: Refusing update from remote peer 0393818808c64e8ab7bbb710b8300d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:16.888259 17908 consensus_queue.cc:1035] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f0a903a0c649b59505c32474cb09d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:16.888495 17905 consensus_queue.cc:1035] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea90775b4b248fab113fe562c530672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:16.897590 17913 mvcc.cc:205] Tried to move back new op lower bound from 7333765516843843584 to 7333765516606599168. Current Snapshot: MvccSnapshot[applied={T|T < 7333765516843843584}]
I20260927 00:48:16.898393 17912 mvcc.cc:205] Tried to move back new op lower bound from 7333765516843843584 to 7333765516606599168. Current Snapshot: MvccSnapshot[applied={T|T < 7333765516843843584}]
I20260927 00:48:16.904280 17419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:16.904417 17419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:16.906558 17419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce6115251cad4649a6fa6ac7046f5a8c: Sending DeleteTablet for 3 replicas of tablet ebda8e7ffa494bf3bbc09c298f4f1498
I20260927 00:48:16.907387 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34630
I20260927 00:48:16.907435 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38756
I20260927 00:48:16.907413 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet ebda8e7ffa494bf3bbc09c298f4f1498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260927 00:48:16.907845 17921 tablet_replica.cc:330] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: stopping tablet replica
I20260927 00:48:16.907989 17922 tablet_replica.cc:330] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7: stopping tablet replica
I20260927 00:48:16.908126 17921 raft_consensus.cc:2229] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:16.908219 17922 raft_consensus.cc:2229] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:16.908430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17483
I20260927 00:48:16.908450 17921 raft_consensus.cc:2258] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:16.908469 17922 raft_consensus.cc:2258] T ebda8e7ffa494bf3bbc09c298f4f1498 P c8f0a903a0c649b59505c32474cb09d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:16.909104 17920 tablet_replica.cc:330] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: stopping tablet replica
I20260927 00:48:16.909955 17920 raft_consensus.cc:2229] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:16.910218 17920 raft_consensus.cc:2258] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:16.910461 17921 ts_tablet_manager.cc:1905] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:16.911496 17920 ts_tablet_manager.cc:1905] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:16.913635 17921 ts_tablet_manager.cc:1918] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:16.913740 17921 log.cc:1196] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/ebda8e7ffa494bf3bbc09c298f4f1498
I20260927 00:48:16.914109 17921 ts_tablet_manager.cc:1939] T ebda8e7ffa494bf3bbc09c298f4f1498 P 0393818808c64e8ab7bbb710b8300d2e: Deleting consensus metadata
I20260927 00:48:16.914220 17920 ts_tablet_manager.cc:1918] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:16.914300 17920 log.cc:1196] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/ebda8e7ffa494bf3bbc09c298f4f1498
I20260927 00:48:16.914588 17920 ts_tablet_manager.cc:1939] T ebda8e7ffa494bf3bbc09c298f4f1498 P cea90775b4b248fab113fe562c530672: Deleting consensus metadata
I20260927 00:48:16.915313 17405 catalog_manager.cc:4841] TS 0393818808c64e8ab7bbb710b8300d2e (127.0.98.2:44217): tablet ebda8e7ffa494bf3bbc09c298f4f1498 (table test-table [id=f820866b6688433c8ffa58c2438f87e9]) successfully deleted
I20260927 00:48:16.915638 17405 catalog_manager.cc:4841] TS cea90775b4b248fab113fe562c530672 (127.0.98.3:38459): tablet ebda8e7ffa494bf3bbc09c298f4f1498 (table test-table [id=f820866b6688433c8ffa58c2438f87e9]) successfully deleted
W20260927 00:48:16.917969 17406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44949 (error 108)
I20260927 00:48:16.918931 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:16.918645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44949 (local address 127.0.0.1:58930)
0927 00:48:16.918802 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:16.918861 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 00:48:16.919078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17617
W20260927 00:48:16.920300 17406 catalog_manager.cc:4584] TS c8f0a903a0c649b59505c32474cb09d7 (127.0.98.1:44949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebda8e7ffa494bf3bbc09c298f4f1498: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44949: connect: Connection refused (error 111)
I20260927 00:48:16.924914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17751
I20260927 00:48:16.930104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17389
2026-09-27T00:48:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2563 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:16Z Disabled control of system clock
I20260927 00:48:16.959049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39493
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:17.066624 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:17.066875 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:17.066910 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:17.070820 17927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:17.070874 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:17.070892 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:17.070909 17927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:17.070925 17927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:17.075563 17927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:17.076597 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:17.077986 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:17.086149 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.086244 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:17.086321 17927 server_base.cc:1034] running on GCE node
W20260927 00:48:17.086153 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:17.086978 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:17.088124 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:17.089308 17927 hybrid_clock.cc:648] HybridClock initialized: now 1790470097089287 us; error 34 us; skew 500 ppm
I20260927 00:48:17.091454 17927 webserver.cc:460] Webserver started at http://127.0.98.62:32919/ using document root <none> and password file <none>
I20260927 00:48:17.092186 17927 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:17.092283 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:17.092511 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:17.094704 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "4606a0d325a54687b11d1846797befaa"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "36891a50388986ba3b84aa1daa4d4b32"
server_key_iv: "d3c4086aea32efdf7f67550fd593f5fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.095265 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "4606a0d325a54687b11d1846797befaa"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "36891a50388986ba3b84aa1daa4d4b32"
server_key_iv: "d3c4086aea32efdf7f67550fd593f5fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.098906 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:17.101189 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.102438 17927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:17.102578 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "4606a0d325a54687b11d1846797befaa"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "36891a50388986ba3b84aa1daa4d4b32"
server_key_iv: "d3c4086aea32efdf7f67550fd593f5fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.102684 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:17.118569 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:17.119257 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:17.119446 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:17.126041 17927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39493
I20260927 00:48:17.126066 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39493 every 8 connection(s)
I20260927 00:48:17.126932 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:17.129868 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:17.134186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 17927
I20260927 00:48:17.134318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:17.134581 392 external_mini_cluster.cc:1428] Setting key 1ca3307a12a3ac9011ae803780676118
I20260927 00:48:17.135432 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: Bootstrap starting.
I20260927 00:48:17.137562 17994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:17.138405 17994 log.cc:824] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:17.140388 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: No bootstrap required, opened a new log
I20260927 00:48:17.144022 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.136036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.0.98.62:39493)
0927 00:48:17.136683 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.136698 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.136732 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:48:17.137343 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.137360 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.137835 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.138138 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:17.138481 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.139659 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.140844 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.141342 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.141459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.142342 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:17.142347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:17.142359 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:17.142730 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:17.142790 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:17.142810 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:17.142883 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.143037 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.143333 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260927 00:48:17.144343 17994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260927 00:48:17.144572 17994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:17.144649 17994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4606a0d325a54687b11d1846797befaa, State: Initialized, Role: FOLLOWER
I20260927 00:48:17.145129 17994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260927 00:48:17.145292 17994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:17.145354 17994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:17.145445 17994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:17.146528 17994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260927 00:48:17.146803 17994 leader_election.cc:302] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4606a0d325a54687b11d1846797befaa; no voters:
I20260927 00:48:17.147037 17994 leader_election.cc:288] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:17.147146 17999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:17.147362 17999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [term 1 LEADER]: Becoming Leader. State: Replica: 4606a0d325a54687b11d1846797befaa, State: Running, Role: LEADER
I20260927 00:48:17.147661 17999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } }
I20260927 00:48:17.148208 17994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:17.149207 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4606a0d325a54687b11d1846797befaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } } }
I20260927 00:48:17.149340 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:17.149725 18008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:17.150511 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4606a0d325a54687b11d1846797befaa. Latest consensus state: current_term: 1 leader_uuid: "4606a0d325a54687b11d1846797befaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606a0d325a54687b11d1846797befaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39493 } } }
I20260927 00:48:17.150607 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:17.152484 18008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:17.157581 18008 catalog_manager.cc:1347] Generated new cluster ID: 9ed422cf5e874cf6864316001a546ad7
I20260927 00:48:17.157646 18008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:17.166474 18008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:17.167531 18008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:17.172804 18008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: Generated new TSK 0
I20260927 00:48:17.173460 18008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:17.177049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--builtin_ntp_servers=127.0.98.20:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:17.287642 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:17.287881 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:17.287919 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:17.291923 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:17.291970 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:17.292058 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:17.296788 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:17.297847 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:17.299226 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:17.308588 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.308758 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.308657 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:17.308861 18018 server_base.cc:1034] running on GCE node
I20260927 00:48:17.309324 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:17.310031 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:17.311283 18018 hybrid_clock.cc:648] HybridClock initialized: now 1790470097311260 us; error 48 us; skew 500 ppm
I20260927 00:48:17.313238 18018 webserver.cc:460] Webserver started at http://127.0.98.1:33683/ using document root <none> and password file <none>
I20260927 00:48:17.313849 18018 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:17.313933 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:17.314157 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:17.316558 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d488e603c9549e29cf56f3a047de5c2"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "c092838fede69b00015b2168760ca91e"
server_key_iv: "1299888ead3d8e01c1dd2b9bc526e28f"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.317118 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d488e603c9549e29cf56f3a047de5c2"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "c092838fede69b00015b2168760ca91e"
server_key_iv: "1299888ead3d8e01c1dd2b9bc526e28f"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.320665 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:17.323027 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.324288 18018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:17.324426 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "5d488e603c9549e29cf56f3a047de5c2"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "c092838fede69b00015b2168760ca91e"
server_key_iv: "1299888ead3d8e01c1dd2b9bc526e28f"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.324536 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:17.358402 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:17.359170 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:17.359365 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:17.359959 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:17.360899 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:17.360967 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.361035 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:17.361086 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.370108 18018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43899
I20260927 00:48:17.370162 18145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43899 every 8 connection(s)
I20260927 00:48:17.371026 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:17.373526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18018
I20260927 00:48:17.373657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:17.373991 392 external_mini_cluster.cc:1428] Setting key eab8a9a5c7ccb12a2b710b425c268334
I20260927 00:48:17.376281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--builtin_ntp_servers=127.0.98.20:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:17.381192 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.372747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49109 (local address 127.0.98.62:39493)
0927 00:48:17.372891 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.372896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.374270 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 00:48:17.375994 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.375996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.376057 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.376119 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:17.377375 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.377970 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.379182 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.379413 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.379526 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.380656 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:17.380665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:17.380667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:17.380679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:17.380698 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:17.380706 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:17.380777 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.380931 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.380993 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:17.382057 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.373295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.1:49109)
0927 00:48:17.374108 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.374162 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.375761 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.376342 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.376352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.376496 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:17.377206 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.377228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.378171 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.378175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.379060 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.379070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.379574 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.379579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.379739 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.380391 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:17.380393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:17.380433 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:17.380494 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:17.380792 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:17.380806 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:17.380811 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.380931 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.381203 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260927 00:48:17.383441 18146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260927 00:48:17.383692 18146 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:17.384199 18146 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260927 00:48:17.385916 17958 ts_manager.cc:194] Registered new tserver with Master: 5d488e603c9549e29cf56f3a047de5c2 (127.0.98.1:43899)
I20260927 00:48:17.387639 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49109
W20260927 00:48:17.487659 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:17.487927 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:17.487965 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:17.492074 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:17.492126 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:17.492183 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:17.496946 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:17.498062 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:17.499388 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:17.509261 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.509289 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.509295 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:17.509421 18149 server_base.cc:1034] running on GCE node
I20260927 00:48:17.509939 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:17.510547 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:17.511680 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790470097511665 us; error 35 us; skew 500 ppm
I20260927 00:48:17.513622 18149 webserver.cc:460] Webserver started at http://127.0.98.2:37897/ using document root <none> and password file <none>
I20260927 00:48:17.514242 18149 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:17.514303 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:17.514506 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:17.516492 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "15d0cf481f1745c39c7ad7076cd630da"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "7ac36c6fcc026af46d88b27e16c8f8a2"
server_key_iv: "8681fea40dc6fe18dafc063df52c1629"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.517035 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "15d0cf481f1745c39c7ad7076cd630da"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "7ac36c6fcc026af46d88b27e16c8f8a2"
server_key_iv: "8681fea40dc6fe18dafc063df52c1629"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.520615 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:17.522964 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.524371 18149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:17.524502 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "15d0cf481f1745c39c7ad7076cd630da"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "7ac36c6fcc026af46d88b27e16c8f8a2"
server_key_iv: "8681fea40dc6fe18dafc063df52c1629"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.524608 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:17.545500 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:17.546234 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:17.546446 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:17.547045 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:17.547959 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:17.548024 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.548090 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:17.548138 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.556763 18149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37065
I20260927 00:48:17.556788 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37065 every 8 connection(s)
I20260927 00:48:17.557672 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:17.562551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18149
I20260927 00:48:17.562674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:17.562964 392 external_mini_cluster.cc:1428] Setting key 50e94645e62840de47a298543ce2d288
I20260927 00:48:17.564796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--builtin_ntp_servers=127.0.98.20:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:17.567269 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.559489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44787 (local address 127.0.98.62:39493)
0927 00:48:17.559631 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.559636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.560675 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 00:48:17.562097 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.562101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.562159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.562236 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:17.563204 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.563846 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.565013 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.565239 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.565297 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.566680 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:17.566684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:17.566688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:17.566701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:17.566730 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:17.566740 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:17.566820 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.566996 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.567062 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:48:17.568166 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.559912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.2:44787)
0927 00:48:17.560527 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.560566 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.561887 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.562393 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.562400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.562480 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:17.563062 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.563078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.563964 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.563968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.564837 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.564868 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.565445 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.565447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.565590 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.566388 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:17.566390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:17.566433 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:17.566500 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:17.566836 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:17.566849 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:17.566855 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.566993 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.567305 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260927 00:48:17.569463 18277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260927 00:48:17.569742 18277 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:17.570421 18277 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260927 00:48:17.571588 17958 ts_manager.cc:194] Registered new tserver with Master: 15d0cf481f1745c39c7ad7076cd630da (127.0.98.2:37065)
I20260927 00:48:17.572166 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44787
W20260927 00:48:17.677330 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:17.677573 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:17.677636 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:17.681699 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:17.681751 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:17.681886 18280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:17.686640 18280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:17.687697 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:17.689043 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:17.698684 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.698770 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:17.698639 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:17.698676 18280 server_base.cc:1034] running on GCE node
I20260927 00:48:17.699258 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:17.699862 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:17.701025 18280 hybrid_clock.cc:648] HybridClock initialized: now 1790470097700995 us; error 42 us; skew 500 ppm
I20260927 00:48:17.702858 18280 webserver.cc:460] Webserver started at http://127.0.98.3:33849/ using document root <none> and password file <none>
I20260927 00:48:17.703455 18280 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:17.703517 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:17.703785 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:17.706106 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "b8d52bee153e49a6b5958e02aeb02f4c"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "a226a781b1c91f231d4b2b3c1f39e865"
server_key_iv: "1d44de9cec3fc7a3d9e239c894ddde9d"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.706749 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b8d52bee153e49a6b5958e02aeb02f4c"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "a226a781b1c91f231d4b2b3c1f39e865"
server_key_iv: "1d44de9cec3fc7a3d9e239c894ddde9d"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.710338 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:48:17.712680 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.713971 18280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:17.714102 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "b8d52bee153e49a6b5958e02aeb02f4c"
format_stamp: "Formatted at 2026-09-27 00:48:17 on dist-test-slave-l5sb"
server_key: "a226a781b1c91f231d4b2b3c1f39e865"
server_key_iv: "1d44de9cec3fc7a3d9e239c894ddde9d"
server_key_version: "encryptionkey@0"
I20260927 00:48:17.714239 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:17.752509 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:17.753238 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:17.753437 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:17.754079 18280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:17.755010 18280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:17.755077 18280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.755146 18280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:17.755185 18280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:17.763695 18280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40269
I20260927 00:48:17.763731 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40269 every 8 connection(s)
I20260927 00:48:17.764583 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:17.771338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18280
I20260927 00:48:17.771459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:17.771799 392 external_mini_cluster.cc:1428] Setting key 880c8dab9be33509376101163513c24f
I20260927 00:48:17.773878 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.766379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40589 (local address 127.0.98.62:39493)
0927 00:48:17.766521 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.766525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.767541 (+ 1016us) server_negotiation.cc:407] Connection header received
0927 00:48:17.768851 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.768853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.768897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.768971 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:17.769862 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.770393 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.771641 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.771977 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.772115 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.773356 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:17.773360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:17.773362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:17.773372 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:17.773391 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:17.773399 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:17.773457 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.773612 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.773693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:48:17.774645 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.766770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39493 (local address 127.0.98.3:40589)
0927 00:48:17.767413 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.767463 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.768674 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.769129 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.769136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.769236 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:17.769698 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.769709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.770519 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.770522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.771506 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.771519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.772150 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.772153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.772381 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.773057 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:17.773061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:17.773113 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:17.773209 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:17.773477 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:17.773485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:17.773491 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.773610 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.773901 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260927 00:48:17.775548 18408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39493
I20260927 00:48:17.775743 18408 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:17.776170 18408 heartbeater.cc:511] Master 127.0.98.62:39493 requested a full tablet report, sending...
I20260927 00:48:17.777033 17958 ts_manager.cc:194] Registered new tserver with Master: b8d52bee153e49a6b5958e02aeb02f4c (127.0.98.3:40269)
I20260927 00:48:17.777527 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40589
I20260927 00:48:17.785507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:17.790683 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.787394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.98.62:39493)
0927 00:48:17.787536 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.787542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.787679 (+ 137us) server_negotiation.cc:407] Connection header received
0927 00:48:17.787777 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.787779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.787817 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.787879 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:17.788076 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.788589 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.789351 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.789553 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.789628 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.790293 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:17.790296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:17.790298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:17.790309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:17.790327 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:17.790335 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:17.790391 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.790457 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.790500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:48:17.793254 17958 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:17.796206 17958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:17.810762 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.804554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.0.98.1:43899)
0927 00:48:17.805873 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.805887 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.805913 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:48:17.805992 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.805997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.806213 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.806657 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:17.806791 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.807879 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.809330 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810020 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.810127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.810315 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.810463 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.810573 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":160,"threads_started":1}
I20260927 00:48:17.810755 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.804403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43899 (local address 127.0.0.1:34772)
0927 00:48:17.805208 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.805258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.805384 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.806375 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.806381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.806404 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:17.806503 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.806509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.807966 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.807970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.809211 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.809221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810144 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.810148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.810256 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.810298 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.810432 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.810560 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":130,"thread_start_us":72,"threads_started":1}
I20260927 00:48:17.811311 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.804725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40269 (local address 127.0.0.1:54694)
0927 00:48:17.805841 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.805854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.805945 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.806446 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.806451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.806473 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:17.806568 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.806573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.807943 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.807949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.809641 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.809652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810888 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.810892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.810987 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.811009 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.811102 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.811186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":309,"thread_start_us":79,"threads_started":1}
I20260927 00:48:17.811517 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.804439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37065 (local address 127.0.0.1:35292)
0927 00:48:17.805468 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.805481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.805567 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.806232 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.806237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.806260 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:17.806421 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.806430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.808185 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.808209 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.809641 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.809652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810704 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.810707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.810797 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.811248 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.811292 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.811345 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":183,"thread_start_us":98,"threads_started":1}
I20260927 00:48:17.811780 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.805093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.0.98.3:40269)
0927 00:48:17.805530 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.805544 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.806061 (+ 517us) server_negotiation.cc:407] Connection header received
0927 00:48:17.806158 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.806162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.806321 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.806458 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:17.806645 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.807802 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.809790 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810547 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.810647 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.810822 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.811522 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.811617 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":142,"threads_started":1}
I20260927 00:48:17.812291 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.805205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.0.98.2:37065)
0927 00:48:17.805682 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.805710 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.805737 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:48:17.805837 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.805843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.806076 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.806246 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:17.806556 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.807821 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.809842 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.810573 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.811809 (+ 1236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.811980 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.812042 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.812139 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":116,"threads_started":1}
I20260927 00:48:17.814222 18080 tablet_service.cc:1467] Processing CreateTablet for tablet e837abeddca641669cea43e07f95cd8b (DEFAULT_TABLE table=test-table [id=090c592dd7964297ae7dd37e0bcdcd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:17.814405 18342 tablet_service.cc:1467] Processing CreateTablet for tablet e837abeddca641669cea43e07f95cd8b (DEFAULT_TABLE table=test-table [id=090c592dd7964297ae7dd37e0bcdcd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:17.814817 18211 tablet_service.cc:1467] Processing CreateTablet for tablet e837abeddca641669cea43e07f95cd8b (DEFAULT_TABLE table=test-table [id=090c592dd7964297ae7dd37e0bcdcd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:17.815296 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e837abeddca641669cea43e07f95cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:17.815428 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e837abeddca641669cea43e07f95cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:17.815685 18211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e837abeddca641669cea43e07f95cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:17.821596 18425 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Bootstrap starting.
I20260927 00:48:17.823089 18426 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Bootstrap starting.
I20260927 00:48:17.823266 18427 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Bootstrap starting.
I20260927 00:48:17.824630 18425 tablet_bootstrap.cc:654] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:17.825196 18426 tablet_bootstrap.cc:654] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:17.825187 18427 tablet_bootstrap.cc:654] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:17.825505 18425 log.cc:824] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:17.826035 18426 log.cc:824] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:17.826035 18427 log.cc:824] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:17.827493 18425 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: No bootstrap required, opened a new log
I20260927 00:48:17.827720 18427 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: No bootstrap required, opened a new log
I20260927 00:48:17.827921 18427 ts_tablet_manager.cc:1397] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:48:17.827702 18425 ts_tablet_manager.cc:1397] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:17.831061 18426 tablet_bootstrap.cc:492] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: No bootstrap required, opened a new log
I20260927 00:48:17.831290 18426 ts_tablet_manager.cc:1397] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:48:17.831444 18425 raft_consensus.cc:348] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.831677 18425 raft_consensus.cc:374] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:17.831727 18425 raft_consensus.cc:729] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d488e603c9549e29cf56f3a047de5c2, State: Initialized, Role: FOLLOWER
I20260927 00:48:17.831861 18427 raft_consensus.cc:348] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.832109 18427 raft_consensus.cc:374] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:17.832170 18427 raft_consensus.cc:729] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d52bee153e49a6b5958e02aeb02f4c, State: Initialized, Role: FOLLOWER
I20260927 00:48:17.832190 18425 consensus_queue.cc:260] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.832616 18427 consensus_queue.cc:260] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.832929 18146 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260927 00:48:17.833256 18425 ts_tablet_manager.cc:1428] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:48:17.833541 18427 ts_tablet_manager.cc:1428] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:48:17.833922 18408 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260927 00:48:17.835664 18426 raft_consensus.cc:348] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.835880 18426 raft_consensus.cc:374] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:17.835937 18426 raft_consensus.cc:729] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d0cf481f1745c39c7ad7076cd630da, State: Initialized, Role: FOLLOWER
I20260927 00:48:17.836383 18426 consensus_queue.cc:260] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.837078 18277 heartbeater.cc:503] Master 127.0.98.62:39493 was elected leader, sending a full tablet report...
I20260927 00:48:17.837239 18426 ts_tablet_manager.cc:1428] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 00:48:17.872278 18147 tablet.cc:2369] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:17.891683 18432 raft_consensus.cc:482] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:17.891867 18432 raft_consensus.cc:504] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.893136 18432 leader_election.cc:288] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d488e603c9549e29cf56f3a047de5c2 (127.0.98.1:43899), 15d0cf481f1745c39c7ad7076cd630da (127.0.98.2:37065)
I20260927 00:48:17.898169 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.893442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33821 (local address 127.0.98.1:43899)
0927 00:48:17.893553 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.893559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.893695 (+ 136us) server_negotiation.cc:407] Connection header received
0927 00:48:17.893806 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.893812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.893865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.893964 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:17.894271 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.895051 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.896874 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.897466 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.897577 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.897636 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.897969 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.898040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:48:17.898170 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.893516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36767 (local address 127.0.98.2:37065)
0927 00:48:17.893642 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.893647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.893898 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:48:17.893996 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.893999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.894049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.894121 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:17.894383 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.895132 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.896874 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.897466 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.897577 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.897636 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.897969 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.898039 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:48:17.898213 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.893358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43899 (local address 127.0.98.3:33821)
0927 00:48:17.893545 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.893560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.893656 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.893989 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.893992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.894011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:17.894138 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.894145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.895210 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.895215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.896750 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.896761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.897636 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.897640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.897812 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.897869 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.897950 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.898046 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":155}
I20260927 00:48:17.898222 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.893407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37065 (local address 127.0.98.3:36767)
0927 00:48:17.893789 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.893820 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:17.893905 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:17.894146 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:17.894149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:17.894164 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:17.894284 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.894289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.895222 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.895224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.896750 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:17.896761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.897636 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.897640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:17.897814 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.897864 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:48:17.897950 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:48:17.898045 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":191,"thread_start_us":135,"threads_started":1}
I20260927 00:48:17.898864 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e837abeddca641669cea43e07f95cd8b" candidate_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d488e603c9549e29cf56f3a047de5c2" is_pre_election: true
I20260927 00:48:17.899032 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e837abeddca641669cea43e07f95cd8b" candidate_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d0cf481f1745c39c7ad7076cd630da" is_pre_election: true
I20260927 00:48:17.899197 18100 raft_consensus.cc:2454] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d52bee153e49a6b5958e02aeb02f4c in term 0.
I20260927 00:48:17.899364 18231 raft_consensus.cc:2454] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d52bee153e49a6b5958e02aeb02f4c in term 0.
I20260927 00:48:17.899757 18297 leader_election.cc:302] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d488e603c9549e29cf56f3a047de5c2, b8d52bee153e49a6b5958e02aeb02f4c; no voters:
I20260927 00:48:17.900032 18432 raft_consensus.cc:2781] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:17.900125 18432 raft_consensus.cc:482] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:17.900179 18432 raft_consensus.cc:3037] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:17.901423 18432 raft_consensus.cc:504] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.901789 18432 leader_election.cc:288] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [CANDIDATE]: Term 1 election: Requested vote from peers 5d488e603c9549e29cf56f3a047de5c2 (127.0.98.1:43899), 15d0cf481f1745c39c7ad7076cd630da (127.0.98.2:37065)
I20260927 00:48:17.902076 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e837abeddca641669cea43e07f95cd8b" candidate_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d488e603c9549e29cf56f3a047de5c2"
I20260927 00:48:17.902138 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e837abeddca641669cea43e07f95cd8b" candidate_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d0cf481f1745c39c7ad7076cd630da"
I20260927 00:48:17.902209 18100 raft_consensus.cc:3037] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:17.902271 18231 raft_consensus.cc:3037] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:17.903271 18100 raft_consensus.cc:2454] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d52bee153e49a6b5958e02aeb02f4c in term 1.
I20260927 00:48:17.903306 18231 raft_consensus.cc:2454] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d52bee153e49a6b5958e02aeb02f4c in term 1.
I20260927 00:48:17.903589 18295 leader_election.cc:302] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d0cf481f1745c39c7ad7076cd630da, b8d52bee153e49a6b5958e02aeb02f4c; no voters:
I20260927 00:48:17.903784 18432 raft_consensus.cc:2781] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:17.904028 18432 raft_consensus.cc:686] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 1 LEADER]: Becoming Leader. State: Replica: b8d52bee153e49a6b5958e02aeb02f4c, State: Running, Role: LEADER
I20260927 00:48:17.904363 18432 consensus_queue.cc:237] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } }
I20260927 00:48:17.907498 17958 catalog_manager.cc:5495] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d52bee153e49a6b5958e02aeb02f4c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d52bee153e49a6b5958e02aeb02f4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40269 } health_report { overall_health: HEALTHY } } }
I20260927 00:48:17.933446 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.929897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.0.98.3:40269)
0927 00:48:17.930039 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:17.930044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:17.930059 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:48:17.930210 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:17.930213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:17.930269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:17.930381 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:17.930616 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.931117 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.931977 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:17.932212 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:17.932329 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:17.932618 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:17.933045 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:17.933181 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:17.933241 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:48:17.933287 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:48:17.939244 18100 raft_consensus.cc:1261] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 1 FOLLOWER]: Refusing update from remote peer b8d52bee153e49a6b5958e02aeb02f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:17.939242 18231 raft_consensus.cc:1261] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 1 FOLLOWER]: Refusing update from remote peer b8d52bee153e49a6b5958e02aeb02f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:17.939968 18435 consensus_queue.cc:1035] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d488e603c9549e29cf56f3a047de5c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:17.940137 18432 consensus_queue.cc:1035] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d0cf481f1745c39c7ad7076cd630da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:17.948203 18439 mvcc.cc:205] Tried to move back new op lower bound from 7333765521151737856 to 7333765521018785792. Current Snapshot: MvccSnapshot[applied={T|T < 7333765521151737856}]
I20260927 00:48:17.948654 18440 mvcc.cc:205] Tried to move back new op lower bound from 7333765521151737856 to 7333765521018785792. Current Snapshot: MvccSnapshot[applied={T|T < 7333765521151737856}]
I20260927 00:48:17.955327 17957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:17.955493 17957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:17.957844 17957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4606a0d325a54687b11d1846797befaa: Sending DeleteTablet for 3 replicas of tablet e837abeddca641669cea43e07f95cd8b
I20260927 00:48:17.958539 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet e837abeddca641669cea43e07f95cd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:17 UTC) from {username='slave'} at 127.0.0.1:34772
I20260927 00:48:17.958770 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet e837abeddca641669cea43e07f95cd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:17 UTC) from {username='slave'} at 127.0.0.1:54694
I20260927 00:48:17.958897 18211 tablet_service.cc:1514] Processing DeleteTablet for tablet e837abeddca641669cea43e07f95cd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:17 UTC) from {username='slave'} at 127.0.0.1:35292
I20260927 00:48:17.959043 18448 tablet_replica.cc:330] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2: stopping tablet replica
I20260927 00:48:17.959343 18448 raft_consensus.cc:2229] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:17.959657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18018
I20260927 00:48:17.959661 18448 raft_consensus.cc:2258] T e837abeddca641669cea43e07f95cd8b P 5d488e603c9549e29cf56f3a047de5c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:17.961119 18449 tablet_replica.cc:330] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: stopping tablet replica
I20260927 00:48:17.961395 18449 raft_consensus.cc:2229] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:17.961731 18449 raft_consensus.cc:2258] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:17.962000 18450 tablet_replica.cc:330] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: stopping tablet replica
I20260927 00:48:17.962263 18450 raft_consensus.cc:2229] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:17.962518 18450 raft_consensus.cc:2258] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:17.963925 18449 ts_tablet_manager.cc:1905] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:17.965020 18450 ts_tablet_manager.cc:1905] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:17.966833 18449 ts_tablet_manager.cc:1918] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:17.966995 18449 log.cc:1196] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/e837abeddca641669cea43e07f95cd8b
W20260927 00:48:17.967247 17944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43899 (error 108)
I20260927 00:48:17.967458 18449 ts_tablet_manager.cc:1939] T e837abeddca641669cea43e07f95cd8b P b8d52bee153e49a6b5958e02aeb02f4c: Deleting consensus metadata
I20260927 00:48:17.967839 18450 ts_tablet_manager.cc:1918] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:17.967935 18450 log.cc:1196] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/e837abeddca641669cea43e07f95cd8b
I20260927 00:48:17.968041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18149
I20260927 00:48:17.968240 18450 ts_tablet_manager.cc:1939] T e837abeddca641669cea43e07f95cd8b P 15d0cf481f1745c39c7ad7076cd630da: Deleting consensus metadata
I20260927 00:48:17.969508 17945 catalog_manager.cc:4841] TS b8d52bee153e49a6b5958e02aeb02f4c (127.0.98.3:40269): tablet e837abeddca641669cea43e07f95cd8b (table test-table [id=090c592dd7964297ae7dd37e0bcdcd8c]) successfully deleted
I20260927 00:48:17.971186 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.970845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43899 (local address 127.0.0.1:34786)
0927 00:48:17.971029 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.971101 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 00:48:17.971436 17944 catalog_manager.cc:4584] TS 5d488e603c9549e29cf56f3a047de5c2 (127.0.98.1:43899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e837abeddca641669cea43e07f95cd8b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43899: connect: Connection refused (error 111)
I20260927 00:48:17.974354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18280
I20260927 00:48:17.975296 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:17.974588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37065 (local address 127.0.0.1:35300)
0927 00:48:17.975170 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:17.975228 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501}
I20260927 00:48:17.980257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 17927
2026-09-27T00:48:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:18 dist-test-slave-l5sb krb5kdc[18454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:18 dist-test-slave-l5sb krb5kdc[18454](info): set up 2 sockets
Sep 27 00:48:18 dist-test-slave-l5sb krb5kdc[18454](info): commencing operation
krb5kdc: starting...
W20260927 00:48:19.017586 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:19.139987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:19.251470 18467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:19.251754 18467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:19.251796 18467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:19.255885 18467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:19.255947 18467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:19.255966 18467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:19.256004 18467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:19.256047 18467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:19.261745 18467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:19.262873 18467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:19.263898 18467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:19.272408 18472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:19.272418 18475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.272526 18467 server_base.cc:1034] running on GCE node
W20260927 00:48:19.272426 18473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.273312 18467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:19.274344 18467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:19.275498 18467 hybrid_clock.cc:648] HybridClock initialized: now 1790470099275470 us; error 41 us; skew 500 ppm
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:19.283851 18467 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:19.284777 18467 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260927 00:48:19.285332 18467 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:19.285418 18467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:19.285642 18467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:19.287706 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "7695467e55b148429191cb1a37c0faf2"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "77cbf422aab85d35720f0afe00600dc6"
server_key_iv: "e7d8900de0b5e349973e6aca4c5eb413"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.288254 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "7695467e55b148429191cb1a37c0faf2"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "77cbf422aab85d35720f0afe00600dc6"
server_key_iv: "e7d8900de0b5e349973e6aca4c5eb413"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.291761 18467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:48:19.294051 18482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.295181 18467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:48:19.295310 18467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "7695467e55b148429191cb1a37c0faf2"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "77cbf422aab85d35720f0afe00600dc6"
server_key_iv: "e7d8900de0b5e349973e6aca4c5eb413"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.295408 18467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:19.314836 18467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:19.318094 18467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:19.318290 18467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:19.325076 18467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33721
I20260927 00:48:19.325091 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33721 every 8 connection(s)
I20260927 00:48:19.326015 18467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:19.328114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18467
I20260927 00:48:19.328256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:19.328547 392 external_mini_cluster.cc:1428] Setting key 5de1de088092771f582520d42a4a27ec
I20260927 00:48:19.328866 18535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:19.336007 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: Bootstrap starting.
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:19.338601 18535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:19.339510 18535 log.cc:824] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:19.341704 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: No bootstrap required, opened a new log
I20260927 00:48:19.343120 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:19.329970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.98.62:33721)
0927 00:48:19.330369 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:19.330388 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:19.330431 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:48:19.331179 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:19.331201 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:19.331608 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:19.331968 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:19.332663 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.334155 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.335019 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.335698 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.335861 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:19.338512 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:19.338518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:19.338531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:19.338559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:19.340292 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.340770 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.340775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.340780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.340898 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.341195 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.341197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.341198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.341599 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:19.341743 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:19.342076 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:48:19.342391 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260927 00:48:19.345854 18535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260927 00:48:19.346079 18535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:19.346140 18535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7695467e55b148429191cb1a37c0faf2, State: Initialized, Role: FOLLOWER
I20260927 00:48:19.346539 18535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260927 00:48:19.346697 18535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:19.346774 18535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:19.346890 18535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:19.347810 18535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260927 00:48:19.348187 18535 leader_election.cc:302] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7695467e55b148429191cb1a37c0faf2; no voters:
I20260927 00:48:19.348568 18535 leader_election.cc:288] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:19.348667 18540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:19.348917 18540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [term 1 LEADER]: Becoming Leader. State: Replica: 7695467e55b148429191cb1a37c0faf2, State: Running, Role: LEADER
I20260927 00:48:19.349215 18540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260927 00:48:19.349932 18535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:19.351553 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7695467e55b148429191cb1a37c0faf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } } }
I20260927 00:48:19.351707 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:19.351951 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7695467e55b148429191cb1a37c0faf2. Latest consensus state: current_term: 1 leader_uuid: "7695467e55b148429191cb1a37c0faf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695467e55b148429191cb1a37c0faf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } } }
I20260927 00:48:19.352059 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:19.352361 18549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:19.355248 18549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:19.360741 18549 catalog_manager.cc:1347] Generated new cluster ID: e5583f577fd84d3c8202a4c0855a8b7f
I20260927 00:48:19.360816 18549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:19.369038 18549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:19.370003 18549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:19.382222 18549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: Generated new TSK 0
I20260927 00:48:19.382877 18549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:48:19.412731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:19.522884 18561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:19.523123 18561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:19.523164 18561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:19.527180 18561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:19.527235 18561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:19.527318 18561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:19.532505 18561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:19.533515 18561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:19.534560 18561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:19.543891 18566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:19.543884 18567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:19.543884 18569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.544348 18561 server_base.cc:1034] running on GCE node
I20260927 00:48:19.544697 18561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:19.545257 18561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:19.546411 18561 hybrid_clock.cc:648] HybridClock initialized: now 1790470099546396 us; error 29 us; skew 500 ppm
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:19.549129 18561 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:48:19.550143 18561 webserver.cc:460] Webserver started at http://127.0.98.1:35075/ using document root <none> and password file <none>
I20260927 00:48:19.550724 18561 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:19.550808 18561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:19.551092 18561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:19.553177 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "52c2ed79b1514aa4aba7f8f87e0d599e"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "3ff32414176337692d8927a880cddda8"
server_key_iv: "fd64bb320c71ce01ec0778c91e37c1fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.553711 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52c2ed79b1514aa4aba7f8f87e0d599e"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "3ff32414176337692d8927a880cddda8"
server_key_iv: "fd64bb320c71ce01ec0778c91e37c1fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.557207 18561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:48:19.559434 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.560582 18561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:48:19.560698 18561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "52c2ed79b1514aa4aba7f8f87e0d599e"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "3ff32414176337692d8927a880cddda8"
server_key_iv: "fd64bb320c71ce01ec0778c91e37c1fe"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.560789 18561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:19.572850 18561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:19.575536 18561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:19.575717 18561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:19.576264 18561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:19.577023 18561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:19.577066 18561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.577128 18561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:19.577190 18561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.585669 18561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45483
I20260927 00:48:19.585691 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45483 every 8 connection(s)
I20260927 00:48:19.586517 18561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:19.588683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18561
I20260927 00:48:19.588809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:19.589093 392 external_mini_cluster.cc:1428] Setting key 15d90e3e3d491d4307a30d82aae7f782
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:48:19.601040 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:19.588214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40483 (local address 127.0.98.62:33721)
0927 00:48:19.588370 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:19.588375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:19.589326 (+ 951us) server_negotiation.cc:407] Connection header received
0927 00:48:19.590424 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:19.590426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:19.590483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:19.590563 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:19.591711 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.592241 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.593399 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.593628 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.593731 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:19.596440 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:19.596447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:19.596453 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:19.596489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:19.598156 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.598861 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.598864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.598866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.598999 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.599598 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.599603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.599605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.599838 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:19.599944 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:19.600676 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 00:48:19.600806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:48:19.602007 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:19.588588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.1:40483)
0927 00:48:19.589136 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:19.589200 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:19.590203 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:19.590774 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:19.590780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:19.591117 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:19.591565 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:19.591577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.592365 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.592368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:19.593266 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:19.593275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.593790 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.593795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:19.593960 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:19.594542 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:19.594557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:19.596276 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:19.598313 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:19.598320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:19.598336 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:19.598687 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:19.599096 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:19.599101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:19.599103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:19.599288 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:19.599943 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:19.599950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:19.600269 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 00:48:19.600542 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 00:48:19.600955 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":161,"threads_started":1}
I20260927 00:48:19.603320 18690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260927 00:48:19.603576 18690 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:19.604107 18690 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260927 00:48:19.605788 18499 ts_manager.cc:194] Registered new tserver with Master: 52c2ed79b1514aa4aba7f8f87e0d599e (127.0.98.1:45483)
I20260927 00:48:19.607493 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:48:19.617159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:19.725069 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:19.725311 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:19.725353 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:19.729453 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:19.729511 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:19.729599 18695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:19.734776 18695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:19.735833 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:19.736830 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:19.746096 18701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:19.746097 18700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.746325 18695 server_base.cc:1034] running on GCE node
W20260927 00:48:19.746124 18703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.746904 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:19.747509 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:19.748687 18695 hybrid_clock.cc:648] HybridClock initialized: now 1790470099748670 us; error 32 us; skew 500 ppm
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:19.751570 18695 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:48:19.752544 18695 webserver.cc:460] Webserver started at http://127.0.98.2:44799/ using document root <none> and password file <none>
I20260927 00:48:19.753087 18695 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:19.753149 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:19.753366 18695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:19.755393 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "f9b202d0e5cb45cf886b099f0f9826b0"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "f5a77f6198c05904e2bcafc534da4ae9"
server_key_iv: "73cf5fb966be8d585ffaae59aaa41d1c"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.755934 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f9b202d0e5cb45cf886b099f0f9826b0"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "f5a77f6198c05904e2bcafc534da4ae9"
server_key_iv: "73cf5fb966be8d585ffaae59aaa41d1c"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.759532 18695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:48:19.761899 18710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.763059 18695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:48:19.763196 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "f9b202d0e5cb45cf886b099f0f9826b0"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "f5a77f6198c05904e2bcafc534da4ae9"
server_key_iv: "73cf5fb966be8d585ffaae59aaa41d1c"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.763332 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:19.786904 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:19.790300 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:19.790500 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:19.791059 18695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:19.791872 18695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:19.791924 18695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.792001 18695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:19.792032 18695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.800589 18695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41853
I20260927 00:48:19.800647 18823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41853 every 8 connection(s)
I20260927 00:48:19.801514 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:19.802982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18695
I20260927 00:48:19.803123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:19.803452 392 external_mini_cluster.cc:1428] Setting key df8d554bb2ea732ec89685ef1ef060c3
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:48:19.815505 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:19.803425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53073 (local address 127.0.98.62:33721)
0927 00:48:19.803590 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:19.803594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:19.804622 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 00:48:19.805855 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:19.805858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:19.805902 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:19.805971 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:19.807402 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.807929 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.809133 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.809359 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.809465 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:19.811849 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:19.811853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:19.811856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:19.811901 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:19.813345 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.813881 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.813884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.813886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.813978 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:19.814380 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:19.814384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:19.814386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:19.814582 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:19.814697 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:19.815276 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 00:48:19.815382 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:48:19.816746 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:19.803953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.2:53073)
0927 00:48:19.804485 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:19.804527 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:19.805575 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:19.806148 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:19.806169 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:19.806698 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:19.807265 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:19.807278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.808050 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.808053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:19.809007 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:19.809016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:19.809506 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:19.809509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:19.809675 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:19.810304 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:19.810320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:19.811695 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:19.813453 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:19.813459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:19.813474 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:19.813728 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:19.814076 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:19.814079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:19.814081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:19.814283 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:19.814727 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:19.814734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:19.815013 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 00:48:19.815224 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 00:48:19.815599 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":156,"threads_started":1}
I20260927 00:48:19.817904 18824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260927 00:48:19.818148 18824 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:19.818619 18824 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260927 00:48:19.819731 18499 ts_manager.cc:194] Registered new tserver with Master: f9b202d0e5cb45cf886b099f0f9826b0 (127.0.98.2:41853)
I20260927 00:48:19.820679 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:48:19.830649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:19.940369 18829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:19.940631 18829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:19.940675 18829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:19.944855 18829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:19.944911 18829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:19.944969 18829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:19.950205 18829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:19.951253 18829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:19.952221 18829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:19.961498 18834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.961660 18829 server_base.cc:1034] running on GCE node
W20260927 00:48:19.961508 18835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:19.961519 18837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:19.962188 18829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:19.962749 18829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:19.963901 18829 hybrid_clock.cc:648] HybridClock initialized: now 1790470099963866 us; error 49 us; skew 500 ppm
Sep 27 00:48:19 dist-test-slave-l5sb krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:19.966665 18829 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:48:19.967576 18829 webserver.cc:460] Webserver started at http://127.0.98.3:44787/ using document root <none> and password file <none>
I20260927 00:48:19.968114 18829 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:19.968197 18829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:19.968415 18829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:19.970429 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "513b243def4a4923a3738e54f1769a47"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "8aa20a6b137a5e0041abb71581274be1"
server_key_iv: "02d483d650392d01cc9b1f084e9b6a80"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.970959 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "513b243def4a4923a3738e54f1769a47"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "8aa20a6b137a5e0041abb71581274be1"
server_key_iv: "02d483d650392d01cc9b1f084e9b6a80"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.974529 18829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:19.976815 18844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:19.978127 18829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:19.978272 18829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "513b243def4a4923a3738e54f1769a47"
format_stamp: "Formatted at 2026-09-27 00:48:19 on dist-test-slave-l5sb"
server_key: "8aa20a6b137a5e0041abb71581274be1"
server_key_iv: "02d483d650392d01cc9b1f084e9b6a80"
server_key_version: "encryptionkey@0"
I20260927 00:48:19.978377 18829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:19.997717 18829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:20.001134 18829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:20.001338 18829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:20.001965 18829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:20.002799 18829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:20.002878 18829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.002951 18829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:20.002990 18829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.012384 18829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46405
I20260927 00:48:20.012408 18957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46405 every 8 connection(s)
I20260927 00:48:20.013322 18829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:20.016759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 18829
I20260927 00:48:20.016896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:20.017243 392 external_mini_cluster.cc:1428] Setting key a08820413950742a6b819d3fab0d61cb
Sep 27 00:48:20 dist-test-slave-l5sb krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:20.026376 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.015205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49739 (local address 127.0.98.62:33721)
0927 00:48:20.015377 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.015382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.016258 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:48:20.017149 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.017152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.017214 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.017321 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:20.018838 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.019431 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.020660 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.020873 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.020950 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.023365 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:20.023369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:20.023371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:20.023394 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:20.024678 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:20.025098 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:20.025100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:20.025101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:20.025177 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:20.025444 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:20.025446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:20.025447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:20.025603 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:20.025677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.026166 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.026262 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:48:20.027136 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.015634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.3:49739)
0927 00:48:20.016096 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.016138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.016936 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.017508 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.017515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.017995 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:20.018700 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.018723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.019560 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.019573 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.020529 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.020539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.021021 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.021024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.021176 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.021829 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:20.021849 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:20.023223 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:20.024774 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:20.024778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:20.024789 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:20.025018 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:20.025248 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:20.025250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:20.025252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:20.025372 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:20.025680 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:20.025684 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:20.025951 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.026104 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.026331 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260927 00:48:20.028210 18958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260927 00:48:20.028407 18958 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:20.028823 18958 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260927 00:48:20.029904 18499 ts_manager.cc:194] Registered new tserver with Master: 513b243def4a4923a3738e54f1769a47 (127.0.98.3:46405)
I20260927 00:48:20.030618 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49739
I20260927 00:48:20.031138 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:20.040002 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.032996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.98.62:33721)
0927 00:48:20.033120 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.033124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.033268 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:48:20.033358 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.033360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.033394 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.033457 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:20.034233 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.034709 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.035463 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.035655 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.035725 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.037123 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:20.037126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:20.037127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:20.037146 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:20.038419 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:20.038862 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:20.038865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:20.038866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:20.038932 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:20.039206 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:20.039209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:20.039210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:20.039378 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:20.039485 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.039729 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.039824 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:48:20.042717 18498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:20.045590 18498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:20.059484 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.053538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45483 (local address 127.0.0.1:40294)
0927 00:48:20.054087 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.054139 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.054288 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.055419 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.055426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.055445 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:20.055595 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.055602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.056745 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.056748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.057848 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.057858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.058928 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.058932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.059052 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.059100 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.059202 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.059316 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":102,"threads_started":1}
I20260927 00:48:20.059489 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.053712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.98.1:45483)
0927 00:48:20.054922 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.054934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.054958 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:48:20.055041 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.055048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.055267 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.055432 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:20.055712 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.056615 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.057973 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.058806 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.058914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.059203 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.059253 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.059339 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":1039,"threads_started":1}
I20260927 00:48:20.060088 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.054411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.3:46405)
0927 00:48:20.054715 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.054729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.055200 (+ 471us) server_negotiation.cc:407] Connection header received
0927 00:48:20.055419 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.055424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.055589 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.055744 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:20.056144 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.057218 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.058413 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.058988 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.059085 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.059213 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.059505 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.059976 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":128,"threads_started":1}
I20260927 00:48:20.060482 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.054013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.0.1:56274)
0927 00:48:20.055095 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.055111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.055265 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.055897 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.055900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.055917 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:20.056013 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.056033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.057348 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.057351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.058300 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.058308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.059270 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.059273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.059358 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.059379 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.060308 (+ 929us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.060376 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":10,"thread_start_us":83,"threads_started":1}
I20260927 00:48:20.062534 18624 tablet_service.cc:1467] Processing CreateTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 (DEFAULT_TABLE table=test-table [id=7e2d96ed07cf42a5a8e1bbee9ff8e75f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:20.062803 18892 tablet_service.cc:1467] Processing CreateTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 (DEFAULT_TABLE table=test-table [id=7e2d96ed07cf42a5a8e1bbee9ff8e75f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:20.063547 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005dd55699f49239de9bfbb93bbd4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:20.063670 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005dd55699f49239de9bfbb93bbd4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:20.066553 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.053720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.0.1:54342)
0927 00:48:20.054992 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.055004 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.055096 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.055722 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.055724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.055740 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:20.055816 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.055820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.057593 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.057696 (+ 103us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.058914 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.058924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.060131 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.060135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.060229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.060250 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.066313 (+ 6063us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.066406 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"thread_start_us":112,"threads_started":1}
I20260927 00:48:20.066658 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.054383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.0.98.2:41853)
0927 00:48:20.054851 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.054863 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.055095 (+ 232us) server_negotiation.cc:407] Connection header received
0927 00:48:20.055207 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.055213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.055411 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.055548 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:20.055916 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.057450 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.059043 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.059827 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.059938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.060056 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.066364 (+ 6308us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.066490 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":150,"threads_started":1}
I20260927 00:48:20.069752 18758 tablet_service.cc:1467] Processing CreateTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 (DEFAULT_TABLE table=test-table [id=7e2d96ed07cf42a5a8e1bbee9ff8e75f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:20.070623 18976 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Bootstrap starting.
I20260927 00:48:20.070881 18758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005dd55699f49239de9bfbb93bbd4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:20.072844 18976 tablet_bootstrap.cc:654] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:20.074090 18976 log.cc:824] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:20.074451 18975 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Bootstrap starting.
I20260927 00:48:20.076236 18976 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: No bootstrap required, opened a new log
I20260927 00:48:20.076427 18976 ts_tablet_manager.cc:1397] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:20.076740 18975 tablet_bootstrap.cc:654] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:20.077844 18978 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Bootstrap starting.
I20260927 00:48:20.077970 18975 log.cc:824] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:20.079993 18978 tablet_bootstrap.cc:654] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:20.080030 18975 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: No bootstrap required, opened a new log
I20260927 00:48:20.080212 18975 ts_tablet_manager.cc:1397] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:20.080214 18976 raft_consensus.cc:348] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.080521 18976 raft_consensus.cc:374] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:20.080586 18976 raft_consensus.cc:729] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513b243def4a4923a3738e54f1769a47, State: Initialized, Role: FOLLOWER
I20260927 00:48:20.081033 18976 consensus_queue.cc:260] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.081300 18978 log.cc:824] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:20.081730 18958 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
I20260927 00:48:20.082055 18976 ts_tablet_manager.cc:1428] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:48:20.083384 18978 tablet_bootstrap.cc:492] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: No bootstrap required, opened a new log
I20260927 00:48:20.083563 18978 ts_tablet_manager.cc:1397] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:20.084362 18975 raft_consensus.cc:348] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.084619 18975 raft_consensus.cc:374] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:20.084692 18975 raft_consensus.cc:729] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c2ed79b1514aa4aba7f8f87e0d599e, State: Initialized, Role: FOLLOWER
I20260927 00:48:20.085135 18975 consensus_queue.cc:260] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.086155 18690 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
I20260927 00:48:20.086580 18975 ts_tablet_manager.cc:1428] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 00:48:20.087509 18691 tablet.cc:2369] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:20.087347 18978 raft_consensus.cc:348] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.087612 18978 raft_consensus.cc:374] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:20.087666 18978 raft_consensus.cc:729] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b202d0e5cb45cf886b099f0f9826b0, State: Initialized, Role: FOLLOWER
I20260927 00:48:20.088081 18978 consensus_queue.cc:260] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.088743 18824 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
I20260927 00:48:20.089079 18978 ts_tablet_manager.cc:1428] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:48:20.193666 18982 raft_consensus.cc:482] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:20.193938 18982 raft_consensus.cc:504] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.195099 18982 leader_election.cc:288] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9b202d0e5cb45cf886b099f0f9826b0 (127.0.98.2:41853), 513b243def4a4923a3738e54f1769a47 (127.0.98.3:46405)
I20260927 00:48:20.199129 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.195504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60231 (local address 127.0.98.3:46405)
0927 00:48:20.195637 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.195642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.195819 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:48:20.195999 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.196004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.196072 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.196130 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:20.196523 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.196996 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198167 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.198571 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198652 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.198730 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.198925 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.199009 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:48:20.199141 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.195372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.98.1:60231)
0927 00:48:20.195722 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.195734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.195859 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.196202 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.196205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.196249 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:20.196390 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.196396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.197104 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.197117 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.198010 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.198019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.198680 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.198786 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.198814 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.198884 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.198964 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":3,"spinlock_wait_cycles":33920,"thread_start_us":95,"threads_started":1}
I20260927 00:48:20.199154 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.195508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57679 (local address 127.0.98.2:41853)
0927 00:48:20.195650 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.195655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.195888 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:48:20.196051 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.196053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.196093 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.196192 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:20.196491 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.196998 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198107 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.198634 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198736 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.198796 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.198986 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.199050 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:48:20.199400 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.195372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.1:57679)
0927 00:48:20.195808 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.195819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.195940 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.196204 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.196206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.196256 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:20.196383 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.196388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.197107 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.197112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.197994 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.198003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.198754 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.198758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.198844 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.198860 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.198997 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.199064 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":96,"spinlock_wait_cycles":16256,"thread_start_us":137,"threads_started":1}
I20260927 00:48:20.200045 18912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005dd55699f49239de9bfbb93bbd4b1" candidate_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513b243def4a4923a3738e54f1769a47" is_pre_election: true
I20260927 00:48:20.200044 18778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005dd55699f49239de9bfbb93bbd4b1" candidate_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" is_pre_election: true
I20260927 00:48:20.200352 18778 raft_consensus.cc:2454] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c2ed79b1514aa4aba7f8f87e0d599e in term 0.
I20260927 00:48:20.200352 18912 raft_consensus.cc:2454] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c2ed79b1514aa4aba7f8f87e0d599e in term 0.
I20260927 00:48:20.200798 18580 leader_election.cc:302] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513b243def4a4923a3738e54f1769a47, 52c2ed79b1514aa4aba7f8f87e0d599e; no voters:
I20260927 00:48:20.201072 18982 raft_consensus.cc:2781] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:20.201174 18982 raft_consensus.cc:482] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:20.201239 18982 raft_consensus.cc:3037] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:20.202381 18982 raft_consensus.cc:504] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.202745 18982 leader_election.cc:288] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [CANDIDATE]: Term 1 election: Requested vote from peers f9b202d0e5cb45cf886b099f0f9826b0 (127.0.98.2:41853), 513b243def4a4923a3738e54f1769a47 (127.0.98.3:46405)
I20260927 00:48:20.203051 18778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005dd55699f49239de9bfbb93bbd4b1" candidate_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b202d0e5cb45cf886b099f0f9826b0"
I20260927 00:48:20.203125 18912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005dd55699f49239de9bfbb93bbd4b1" candidate_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513b243def4a4923a3738e54f1769a47"
I20260927 00:48:20.203182 18778 raft_consensus.cc:3037] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:20.203224 18912 raft_consensus.cc:3037] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:20.204504 18912 raft_consensus.cc:2454] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c2ed79b1514aa4aba7f8f87e0d599e in term 1.
I20260927 00:48:20.204504 18778 raft_consensus.cc:2454] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c2ed79b1514aa4aba7f8f87e0d599e in term 1.
I20260927 00:48:20.204810 18578 leader_election.cc:302] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c2ed79b1514aa4aba7f8f87e0d599e, f9b202d0e5cb45cf886b099f0f9826b0; no voters:
I20260927 00:48:20.204977 18982 raft_consensus.cc:2781] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:20.205188 18982 raft_consensus.cc:686] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 1 LEADER]: Becoming Leader. State: Replica: 52c2ed79b1514aa4aba7f8f87e0d599e, State: Running, Role: LEADER
I20260927 00:48:20.205494 18982 consensus_queue.cc:237] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } }
I20260927 00:48:20.208628 18498 catalog_manager.cc:5495] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c2ed79b1514aa4aba7f8f87e0d599e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c2ed79b1514aa4aba7f8f87e0d599e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } health_report { overall_health: UNKNOWN } } }
I20260927 00:48:20.253324 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.249307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.98.1:45483)
0927 00:48:20.249459 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.249463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.249479 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:48:20.249641 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.249643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.249701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.249867 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:20.250216 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.250823 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.251821 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.252049 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.252127 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.252451 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:20.252854 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:20.252970 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.253114 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.253158 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:48:20.259018 18912 raft_consensus.cc:1261] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 1 FOLLOWER]: Refusing update from remote peer 52c2ed79b1514aa4aba7f8f87e0d599e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:20.259019 18778 raft_consensus.cc:1261] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 1 FOLLOWER]: Refusing update from remote peer 52c2ed79b1514aa4aba7f8f87e0d599e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:20.259730 18986 consensus_queue.cc:1035] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b202d0e5cb45cf886b099f0f9826b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:20.259889 18982 consensus_queue.cc:1035] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [LEADER]: Connected to new peer: Peer: permanent_uuid: "513b243def4a4923a3738e54f1769a47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:48:20.264287 18959 tablet.cc:2369] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:20.268157 18990 mvcc.cc:205] Tried to move back new op lower bound from 7333765530653945856 to 7333765530444541952. Current Snapshot: MvccSnapshot[applied={T|T < 7333765530653945856}]
I20260927 00:48:20.274482 18498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:20.274633 18498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:20.276703 18498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7695467e55b148429191cb1a37c0faf2: Sending DeleteTablet for 3 replicas of tablet 8005dd55699f49239de9bfbb93bbd4b1
I20260927 00:48:20.277388 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40294
I20260927 00:48:20.277426 18758 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54342
I20260927 00:48:20.277545 18892 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005dd55699f49239de9bfbb93bbd4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56274
I20260927 00:48:20.278013 19000 tablet_replica.cc:330] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: stopping tablet replica
I20260927 00:48:20.278060 18999 tablet_replica.cc:330] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e: stopping tablet replica
I20260927 00:48:20.278280 19001 tablet_replica.cc:330] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: stopping tablet replica
I20260927 00:48:20.278345 18999 raft_consensus.cc:2229] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:20.278631 19001 raft_consensus.cc:2229] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:20.278658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18561
I20260927 00:48:20.278723 18999 raft_consensus.cc:2258] T 8005dd55699f49239de9bfbb93bbd4b1 P 52c2ed79b1514aa4aba7f8f87e0d599e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:20.279268 19000 raft_consensus.cc:2229] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:20.279546 19000 raft_consensus.cc:2258] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:20.279953 19001 raft_consensus.cc:2258] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:20.281105 19000 ts_tablet_manager.cc:1905] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:20.281313 19001 ts_tablet_manager.cc:1905] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:20.283823 19001 ts_tablet_manager.cc:1918] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:20.283932 19001 log.cc:1196] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/8005dd55699f49239de9bfbb93bbd4b1
I20260927 00:48:20.284220 19001 ts_tablet_manager.cc:1939] T 8005dd55699f49239de9bfbb93bbd4b1 P 513b243def4a4923a3738e54f1769a47: Deleting consensus metadata
I20260927 00:48:20.284700 19000 ts_tablet_manager.cc:1918] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:20.284801 19000 log.cc:1196] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/8005dd55699f49239de9bfbb93bbd4b1
I20260927 00:48:20.285131 19000 ts_tablet_manager.cc:1939] T 8005dd55699f49239de9bfbb93bbd4b1 P f9b202d0e5cb45cf886b099f0f9826b0: Deleting consensus metadata
I20260927 00:48:20.285383 18486 catalog_manager.cc:4841] TS 513b243def4a4923a3738e54f1769a47 (127.0.98.3:46405): tablet 8005dd55699f49239de9bfbb93bbd4b1 (table test-table [id=7e2d96ed07cf42a5a8e1bbee9ff8e75f]) successfully deleted
I20260927 00:48:20.286293 18484 catalog_manager.cc:4841] TS f9b202d0e5cb45cf886b099f0f9826b0 (127.0.98.2:41853): tablet 8005dd55699f49239de9bfbb93bbd4b1 (table test-table [id=7e2d96ed07cf42a5a8e1bbee9ff8e75f]) successfully deleted
W20260927 00:48:20.287029 18485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45483 (error 108)
I20260927 00:48:20.287261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18695
I20260927 00:48:20.288642 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.288379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45483 (local address 127.0.0.1:40308)
0927 00:48:20.288505 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.288566 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:48:20.288825 18485 catalog_manager.cc:4584] TS 52c2ed79b1514aa4aba7f8f87e0d599e (127.0.98.1:45483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8005dd55699f49239de9bfbb93bbd4b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45483: connect: Connection refused (error 111)
I20260927 00:48:20.292855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18829
I20260927 00:48:20.298338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 18467
2026-09-27T00:48:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:20Z Disabled control of system clock
I20260927 00:48:20.327427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33895
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:20.434814 19006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:20.435060 19006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:20.435093 19006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:20.438930 19006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:20.438974 19006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:20.438989 19006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:20.439005 19006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:20.439020 19006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:20.443542 19006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:20.444482 19006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:20.445761 19006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:20.453879 19014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:20.453914 19011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:20.454140 19012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:20.454021 19006 server_base.cc:1034] running on GCE node
I20260927 00:48:20.455197 19006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:20.456683 19006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:20.457827 19006 hybrid_clock.cc:648] HybridClock initialized: now 1790470100457799 us; error 40 us; skew 500 ppm
I20260927 00:48:20.460220 19006 webserver.cc:460] Webserver started at http://127.0.98.62:33769/ using document root <none> and password file <none>
I20260927 00:48:20.460906 19006 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:20.461007 19006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:20.461213 19006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:20.463264 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "29b79f4d734495d1e0626ba030737118"
server_key_iv: "47fe3962d4b0f38043f6789a4fbd4ef2"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.463817 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "29b79f4d734495d1e0626ba030737118"
server_key_iv: "47fe3962d4b0f38043f6789a4fbd4ef2"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.467303 19006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:48:20.469548 19020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.470716 19006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:20.470846 19006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "29b79f4d734495d1e0626ba030737118"
server_key_iv: "47fe3962d4b0f38043f6789a4fbd4ef2"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.470949 19006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:20.482180 19006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:20.482839 19006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:20.483026 19006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:20.489619 19006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33895
I20260927 00:48:20.489694 19072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33895 every 8 connection(s)
I20260927 00:48:20.490598 19006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:20.492470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19006
I20260927 00:48:20.492601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:20.492923 392 external_mini_cluster.cc:1428] Setting key 039db567596ebffbca48418a1a595b32
I20260927 00:48:20.494071 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:20.499408 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: Bootstrap starting.
I20260927 00:48:20.501664 19073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:20.501732 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.494338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.98.62:33895)
0927 00:48:20.494924 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.494938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.494971 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:48:20.495613 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.495628 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.495999 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.496346 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:20.496749 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.497693 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.498554 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.499010 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.499186 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.500104 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:20.500109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:20.500121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:20.500489 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:20.500541 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:20.500560 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:20.500664 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.500811 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.501054 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260927 00:48:20.502592 19073 log.cc:824] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:20.504449 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: No bootstrap required, opened a new log
I20260927 00:48:20.507761 19073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } }
I20260927 00:48:20.507962 19073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:20.508028 19073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b8f90a7f64fe3bcb5ce171e19fd8a, State: Initialized, Role: FOLLOWER
I20260927 00:48:20.508419 19073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } }
I20260927 00:48:20.508551 19073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:20.508625 19073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:20.508733 19073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:20.509720 19073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } }
I20260927 00:48:20.510069 19073 leader_election.cc:302] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842b8f90a7f64fe3bcb5ce171e19fd8a; no voters:
I20260927 00:48:20.510357 19073 leader_election.cc:288] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:20.510466 19078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:20.510691 19078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [term 1 LEADER]: Becoming Leader. State: Replica: 842b8f90a7f64fe3bcb5ce171e19fd8a, State: Running, Role: LEADER
I20260927 00:48:20.510986 19078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } }
I20260927 00:48:20.511485 19073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:20.512778 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } } }
I20260927 00:48:20.512784 19080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842b8f90a7f64fe3bcb5ce171e19fd8a. Latest consensus state: current_term: 1 leader_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b8f90a7f64fe3bcb5ce171e19fd8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33895 } } }
I20260927 00:48:20.512907 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:20.513276 19087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:20.512907 19080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:20.516888 19087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:20.522401 19087 catalog_manager.cc:1347] Generated new cluster ID: eb3fda8ba7834bcda941b54fcf06301d
I20260927 00:48:20.522462 19087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:20.538931 19087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:20.540228 19087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:20.548233 19087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: Generated new TSK 0
I20260927 00:48:20.549048 19087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:20.552306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--builtin_ntp_servers=127.0.98.20:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:20.663097 19097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:20.663345 19097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:20.663411 19097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:20.667333 19097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:20.667383 19097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:20.667471 19097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:20.672134 19097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:20.673168 19097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:20.674580 19097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:20.684051 19102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:20.684183 19097 server_base.cc:1034] running on GCE node
W20260927 00:48:20.684064 19103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:20.684051 19105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:20.684671 19097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:20.685302 19097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:20.686470 19097 hybrid_clock.cc:648] HybridClock initialized: now 1790470100686441 us; error 40 us; skew 500 ppm
I20260927 00:48:20.688398 19097 webserver.cc:460] Webserver started at http://127.0.98.1:46267/ using document root <none> and password file <none>
I20260927 00:48:20.688961 19097 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:20.689086 19097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:20.689311 19097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:20.691385 19097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "a39b62365dd041aa94433645d8dd16e1"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "0c6015d9ee5802a903260541a52f62e4"
server_key_iv: "79c6658ddda6e5a85b329178350f846a"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.691933 19097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a39b62365dd041aa94433645d8dd16e1"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "0c6015d9ee5802a903260541a52f62e4"
server_key_iv: "79c6658ddda6e5a85b329178350f846a"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.695604 19097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:20.698108 19111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.699347 19097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:48:20.699478 19097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "a39b62365dd041aa94433645d8dd16e1"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "0c6015d9ee5802a903260541a52f62e4"
server_key_iv: "79c6658ddda6e5a85b329178350f846a"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.699599 19097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:20.712638 19097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:20.713322 19097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:20.713501 19097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:20.714107 19097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:20.715008 19097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:20.715075 19097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.715150 19097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:20.715197 19097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.723873 19097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45647
I20260927 00:48:20.723903 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45647 every 8 connection(s)
I20260927 00:48:20.724769 19097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:20.728322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19097
I20260927 00:48:20.728466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:20.728832 392 external_mini_cluster.cc:1428] Setting key 264a3ff3c4722883290c2f6b8f0548ce
I20260927 00:48:20.731046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--builtin_ntp_servers=127.0.98.20:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:20.734390 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.726604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60027 (local address 127.0.98.62:33895)
0927 00:48:20.726766 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.726769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.727882 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 00:48:20.729458 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.729462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.729523 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.729617 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:20.730680 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.731313 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.732448 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.732691 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.732799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.733907 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:20.733910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:20.733912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:20.733923 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:20.733942 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:20.733950 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:20.734021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.734177 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.734247 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:48:20.735491 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.727050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33895 (local address 127.0.98.1:60027)
0927 00:48:20.727735 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.727774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.729248 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.729845 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.729852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.729937 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:20.730517 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.730534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.731434 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.731437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.732324 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.732333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.732837 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.732841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.733007 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.733620 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:20.733621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:20.733672 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:20.733734 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:20.734035 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:20.734045 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:20.734051 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.734173 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.734661 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260927 00:48:20.736774 19225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33895
I20260927 00:48:20.737076 19225 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:20.737885 19225 heartbeater.cc:511] Master 127.0.98.62:33895 requested a full tablet report, sending...
I20260927 00:48:20.739928 19037 ts_manager.cc:194] Registered new tserver with Master: a39b62365dd041aa94433645d8dd16e1 (127.0.98.1:45647)
I20260927 00:48:20.741746 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60027
W20260927 00:48:20.843830 19228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:20.844067 19228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:20.844105 19228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:20.847929 19228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:20.847973 19228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:20.848030 19228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:20.852476 19228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:20.853427 19228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:20.854660 19228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:20.863735 19236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:20.863807 19234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:20.863749 19233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:20.864059 19228 server_base.cc:1034] running on GCE node
I20260927 00:48:20.864454 19228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:20.865029 19228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:20.866189 19228 hybrid_clock.cc:648] HybridClock initialized: now 1790470100866169 us; error 30 us; skew 500 ppm
I20260927 00:48:20.867949 19228 webserver.cc:460] Webserver started at http://127.0.98.2:46403/ using document root <none> and password file <none>
I20260927 00:48:20.868480 19228 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:20.868539 19228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:20.868772 19228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:20.870826 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "e9b18a230b054134934ff12259656566"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "f92c362e4057454d501e01c489246f4c"
server_key_iv: "51c907fcb26c73ec449300e1b80a02b3"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.871362 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e9b18a230b054134934ff12259656566"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "f92c362e4057454d501e01c489246f4c"
server_key_iv: "51c907fcb26c73ec449300e1b80a02b3"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.874924 19228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:20.877329 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.878605 19228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:20.878751 19228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "e9b18a230b054134934ff12259656566"
format_stamp: "Formatted at 2026-09-27 00:48:20 on dist-test-slave-l5sb"
server_key: "f92c362e4057454d501e01c489246f4c"
server_key_iv: "51c907fcb26c73ec449300e1b80a02b3"
server_key_version: "encryptionkey@0"
I20260927 00:48:20.878857 19228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:20.894207 19228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:20.894876 19228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:20.895059 19228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:20.895625 19228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:20.896582 19228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:20.896657 19228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.896741 19228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:20.896786 19228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:20.905568 19228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46393
I20260927 00:48:20.905587 19355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46393 every 8 connection(s)
I20260927 00:48:20.906505 19228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:20.907584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19228
I20260927 00:48:20.907711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:20.907989 392 external_mini_cluster.cc:1428] Setting key d3061c046a7d6f677a342beea30e4566
I20260927 00:48:20.910094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--builtin_ntp_servers=127.0.98.20:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:20.916849 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.908358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40687 (local address 127.0.98.62:33895)
0927 00:48:20.908515 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:20.908518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:20.909588 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 00:48:20.911050 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:20.911056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:20.911119 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:20.911241 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:20.912103 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.912853 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.914763 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.914975 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.915058 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.916393 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:20.916396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:20.916398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:20.916410 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:20.916426 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:20.916433 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:20.916490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:20.916683 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 00:48:20.916738 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:48:20.918145 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:20.908750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33895 (local address 127.0.98.2:40687)
0927 00:48:20.909447 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:20.909497 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:20.910849 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:20.911400 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:20.911407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:20.911489 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:20.911963 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.911975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.912975 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.912982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.914628 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:20.914655 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:20.915131 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:20.915139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:20.915381 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:20.916057 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:20.916061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:20.916121 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:20.916212 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:20.916516 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:20.916528 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:20.916535 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:48:20.916686 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 00:48:20.917003 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
I20260927 00:48:20.919341 19356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33895
I20260927 00:48:20.919564 19356 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:20.920099 19356 heartbeater.cc:511] Master 127.0.98.62:33895 requested a full tablet report, sending...
I20260927 00:48:20.921116 19037 ts_manager.cc:194] Registered new tserver with Master: e9b18a230b054134934ff12259656566 (127.0.98.2:46393)
I20260927 00:48:20.921669 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40687
W20260927 00:48:21.021462 19359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:21.021710 19359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:21.021777 19359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:21.025727 19359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:21.025818 19359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:21.025913 19359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:21.030566 19359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:21.031622 19359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:21.032944 19359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:21.042402 19367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:21.042454 19359 server_base.cc:1034] running on GCE node
W20260927 00:48:21.042454 19364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:21.042748 19365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:21.043112 19359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:21.043691 19359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:21.044839 19359 hybrid_clock.cc:648] HybridClock initialized: now 1790470101044825 us; error 24 us; skew 500 ppm
I20260927 00:48:21.046645 19359 webserver.cc:460] Webserver started at http://127.0.98.3:32889/ using document root <none> and password file <none>
I20260927 00:48:21.047144 19359 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:21.047200 19359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:21.047351 19359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:21.049321 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "7f8a6cb68af646939b44618996f300b6"
format_stamp: "Formatted at 2026-09-27 00:48:21 on dist-test-slave-l5sb"
server_key: "faa9c1e801f41c9e4c80926cca5e80c2"
server_key_iv: "cc79dfdc6c687818b1c162336ae27a33"
server_key_version: "encryptionkey@0"
I20260927 00:48:21.049856 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7f8a6cb68af646939b44618996f300b6"
format_stamp: "Formatted at 2026-09-27 00:48:21 on dist-test-slave-l5sb"
server_key: "faa9c1e801f41c9e4c80926cca5e80c2"
server_key_iv: "cc79dfdc6c687818b1c162336ae27a33"
server_key_version: "encryptionkey@0"
I20260927 00:48:21.053205 19359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:48:21.055438 19373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:21.056613 19359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:48:21.056699 19359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "7f8a6cb68af646939b44618996f300b6"
format_stamp: "Formatted at 2026-09-27 00:48:21 on dist-test-slave-l5sb"
server_key: "faa9c1e801f41c9e4c80926cca5e80c2"
server_key_iv: "cc79dfdc6c687818b1c162336ae27a33"
server_key_version: "encryptionkey@0"
I20260927 00:48:21.056772 19359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:21.074651 19359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:21.075314 19359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:21.075459 19359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:21.076036 19359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:21.076952 19359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:21.076996 19359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:21.077034 19359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:21.077057 19359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:21.085561 19359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44201
I20260927 00:48:21.085631 19486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44201 every 8 connection(s)
I20260927 00:48:21.086566 19359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:21.095659 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.088356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36755 (local address 127.0.98.62:33895)
0927 00:48:21.088482 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.088485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.089515 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 00:48:21.090961 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.090964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.091024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.091110 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:21.092026 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.092566 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.093683 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.093963 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.094072 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.095179 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:21.095182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:21.095190 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:21.095201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:21.095218 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:21.095225 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:21.095295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.095446 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.095505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:48:21.096493 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.088712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33895 (local address 127.0.98.3:36755)
0927 00:48:21.089373 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.089412 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.090752 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.091305 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.091312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.091398 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:21.091866 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.091878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.092685 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.092689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.093564 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.093572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.094117 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.094120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.094279 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.094905 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:21.094907 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:21.094947 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:21.095009 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:21.095308 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:21.095317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:21.095322 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.095446 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.095716 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":144,"threads_started":1}
I20260927 00:48:21.096750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19359
I20260927 00:48:21.096866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:21.097223 392 external_mini_cluster.cc:1428] Setting key d083ebc22bde36b466aab846e074aae8
I20260927 00:48:21.097659 19487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33895
I20260927 00:48:21.097908 19487 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:21.098390 19487 heartbeater.cc:511] Master 127.0.98.62:33895 requested a full tablet report, sending...
I20260927 00:48:21.099335 19037 ts_manager.cc:194] Registered new tserver with Master: 7f8a6cb68af646939b44618996f300b6 (127.0.98.3:44201)
I20260927 00:48:21.099834 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36755
I20260927 00:48:21.110913 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:21.116751 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.112873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.0.98.62:33895)
0927 00:48:21.112974 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.112978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.113163 (+ 185us) server_negotiation.cc:407] Connection header received
0927 00:48:21.113245 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.113247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.113329 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.113398 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:21.113870 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.114403 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.115265 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.115475 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.115567 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.116378 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:21.116381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:21.116383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:21.116394 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:21.116411 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:21.116418 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:21.116513 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.116615 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.116643 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:48:21.119393 19037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:21.122323 19037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:21.136353 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.130382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46393 (local address 127.0.0.1:42298)
0927 00:48:21.131512 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.131526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.131604 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.132313 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.132318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.132334 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:21.132553 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.132562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133900 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.133903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.135067 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.135074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.135836 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.135942 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.135976 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.136083 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.136199 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":99,"thread_start_us":93,"threads_started":1}
I20260927 00:48:21.136368 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.130975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.0.98.2:46393)
0927 00:48:21.131875 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.131887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.131910 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:48:21.131987 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.131991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.132166 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.132469 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:21.132657 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133762 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135182 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.135718 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135837 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.136054 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.136127 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.136222 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":140,"threads_started":1}
I20260927 00:48:21.136817 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.130465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.0.98.3:44201)
0927 00:48:21.131664 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.131675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.131699 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:48:21.131783 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.131789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.131994 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.132158 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:21.132559 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133762 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135129 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.135853 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135957 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.136140 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.136568 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.136663 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":156,"threads_started":1}
I20260927 00:48:21.136817 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.131079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.98.1:45647)
0927 00:48:21.131565 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.131577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.131906 (+ 329us) server_negotiation.cc:407] Connection header received
0927 00:48:21.131981 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.131986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.132149 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.132273 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:21.132709 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133429 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135061 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.135853 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.135957 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.136149 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.136550 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.136649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":104,"threads_started":1}
I20260927 00:48:21.136714 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.130635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.0.1:47420)
0927 00:48:21.131719 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.131733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.131817 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.132405 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.132407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.132442 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:21.132579 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.132587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133540 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.133546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.134935 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.134945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.136207 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.136211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.136339 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.136397 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.136467 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.136600 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":92,"thread_start_us":85,"threads_started":1}
I20260927 00:48:21.140015 19290 tablet_service.cc:1467] Processing CreateTablet for tablet c314d29452de4758bfff5d0493a21c21 (DEFAULT_TABLE table=test-table [id=85cf35a0dea44245b0c7ae345975c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:21.140013 19159 tablet_service.cc:1467] Processing CreateTablet for tablet c314d29452de4758bfff5d0493a21c21 (DEFAULT_TABLE table=test-table [id=85cf35a0dea44245b0c7ae345975c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:21.140724 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.130329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44201 (local address 127.0.0.1:53682)
0927 00:48:21.131070 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.131116 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.131256 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.132166 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.132172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.132195 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:21.132432 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.132439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.133936 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.133940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.134973 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.134983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.136207 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.136211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.136367 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.136399 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.136467 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.137028 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":582,"thread_start_us":79,"threads_started":1}
I20260927 00:48:21.141134 19159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c314d29452de4758bfff5d0493a21c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:21.141134 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c314d29452de4758bfff5d0493a21c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:21.143253 19421 tablet_service.cc:1467] Processing CreateTablet for tablet c314d29452de4758bfff5d0493a21c21 (DEFAULT_TABLE table=test-table [id=85cf35a0dea44245b0c7ae345975c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:21.144070 19421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c314d29452de4758bfff5d0493a21c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:21.148449 19504 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Bootstrap starting.
I20260927 00:48:21.148861 19505 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Bootstrap starting.
I20260927 00:48:21.149959 19506 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Bootstrap starting.
I20260927 00:48:21.151124 19505 tablet_bootstrap.cc:654] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:21.151805 19506 tablet_bootstrap.cc:654] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:21.151805 19504 tablet_bootstrap.cc:654] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:21.151922 19505 log.cc:824] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:21.152596 19506 log.cc:824] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:21.152596 19504 log.cc:824] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:21.153684 19505 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: No bootstrap required, opened a new log
I20260927 00:48:21.153904 19505 ts_tablet_manager.cc:1397] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260927 00:48:21.154465 19506 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: No bootstrap required, opened a new log
I20260927 00:48:21.154676 19504 tablet_bootstrap.cc:492] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: No bootstrap required, opened a new log
I20260927 00:48:21.154670 19506 ts_tablet_manager.cc:1397] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:48:21.154830 19504 ts_tablet_manager.cc:1397] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:48:21.157037 19505 raft_consensus.cc:348] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.157269 19505 raft_consensus.cc:374] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:21.157323 19505 raft_consensus.cc:729] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39b62365dd041aa94433645d8dd16e1, State: Initialized, Role: FOLLOWER
I20260927 00:48:21.157742 19505 consensus_queue.cc:260] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.158378 19225 heartbeater.cc:503] Master 127.0.98.62:33895 was elected leader, sending a full tablet report...
I20260927 00:48:21.158876 19506 raft_consensus.cc:348] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.158876 19504 raft_consensus.cc:348] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.159101 19504 raft_consensus.cc:374] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:21.159101 19506 raft_consensus.cc:374] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:21.159157 19506 raft_consensus.cc:729] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8a6cb68af646939b44618996f300b6, State: Initialized, Role: FOLLOWER
I20260927 00:48:21.159165 19504 raft_consensus.cc:729] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b18a230b054134934ff12259656566, State: Initialized, Role: FOLLOWER
I20260927 00:48:21.159588 19505 ts_tablet_manager.cc:1428] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:48:21.159596 19506 consensus_queue.cc:260] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.159596 19504 consensus_queue.cc:260] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.160426 19487 heartbeater.cc:503] Master 127.0.98.62:33895 was elected leader, sending a full tablet report...
I20260927 00:48:21.160521 19504 ts_tablet_manager.cc:1428] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:48:21.160597 19506 ts_tablet_manager.cc:1428] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:48:21.160816 19356 heartbeater.cc:503] Master 127.0.98.62:33895 was elected leader, sending a full tablet report...
W20260927 00:48:21.225896 19226 tablet.cc:2369] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:21.294232 19511 raft_consensus.cc:482] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:21.294430 19511 raft_consensus.cc:504] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.295513 19511 leader_election.cc:288] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9b18a230b054134934ff12259656566 (127.0.98.2:46393), a39b62365dd041aa94433645d8dd16e1 (127.0.98.1:45647)
I20260927 00:48:21.299206 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.295771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46393 (local address 127.0.98.3:56131)
0927 00:48:21.295993 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.296019 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.296123 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.296422 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.296425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.296443 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:21.296561 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.296567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.297279 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.297283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.298163 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.298172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.298783 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.298886 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.298910 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.298994 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.299071 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":19}
I20260927 00:48:21.299206 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.295885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56131 (local address 127.0.98.2:46393)
0927 00:48:21.296063 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.296068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.296086 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:48:21.296248 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.296250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.296298 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.296396 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:21.296674 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.297166 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298278 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.298665 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.298836 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.299009 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.299074 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:48:21.299396 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.295879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37165 (local address 127.0.98.1:45647)
0927 00:48:21.296057 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.296063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.296372 (+ 309us) server_negotiation.cc:407] Connection header received
0927 00:48:21.296512 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.296514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.296557 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.296636 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:21.296895 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.297373 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298495 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.298872 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298995 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.299052 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.299222 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.299285 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:48:21.299410 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.295771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.98.3:37165)
0927 00:48:21.296283 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.296293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:21.296397 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:21.296647 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:21.296650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:21.296665 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:21.296789 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.296794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.297478 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.297483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.298375 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:21.298384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.298987 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.298990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:21.299092 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.299116 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:48:21.299205 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:48:21.299271 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":70,"thread_start_us":163,"threads_started":1}
I20260927 00:48:21.300099 19179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c314d29452de4758bfff5d0493a21c21" candidate_uuid: "7f8a6cb68af646939b44618996f300b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39b62365dd041aa94433645d8dd16e1" is_pre_election: true
I20260927 00:48:21.300153 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c314d29452de4758bfff5d0493a21c21" candidate_uuid: "7f8a6cb68af646939b44618996f300b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b18a230b054134934ff12259656566" is_pre_election: true
I20260927 00:48:21.300437 19310 raft_consensus.cc:2454] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8a6cb68af646939b44618996f300b6 in term 0.
I20260927 00:48:21.300437 19179 raft_consensus.cc:2454] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8a6cb68af646939b44618996f300b6 in term 0.
I20260927 00:48:21.300967 19374 leader_election.cc:302] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7f8a6cb68af646939b44618996f300b6, a39b62365dd041aa94433645d8dd16e1, e9b18a230b054134934ff12259656566; no voters:
I20260927 00:48:21.301276 19511 raft_consensus.cc:2781] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:21.301373 19511 raft_consensus.cc:482] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:21.301446 19511 raft_consensus.cc:3037] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:21.302532 19511 raft_consensus.cc:504] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.302910 19511 leader_election.cc:288] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [CANDIDATE]: Term 1 election: Requested vote from peers e9b18a230b054134934ff12259656566 (127.0.98.2:46393), a39b62365dd041aa94433645d8dd16e1 (127.0.98.1:45647)
I20260927 00:48:21.303294 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c314d29452de4758bfff5d0493a21c21" candidate_uuid: "7f8a6cb68af646939b44618996f300b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b18a230b054134934ff12259656566"
I20260927 00:48:21.303303 19179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c314d29452de4758bfff5d0493a21c21" candidate_uuid: "7f8a6cb68af646939b44618996f300b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39b62365dd041aa94433645d8dd16e1"
I20260927 00:48:21.303434 19310 raft_consensus.cc:3037] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:21.303447 19179 raft_consensus.cc:3037] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:21.304481 19310 raft_consensus.cc:2454] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8a6cb68af646939b44618996f300b6 in term 1.
I20260927 00:48:21.304603 19179 raft_consensus.cc:2454] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8a6cb68af646939b44618996f300b6 in term 1.
I20260927 00:48:21.304821 19374 leader_election.cc:302] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8a6cb68af646939b44618996f300b6, e9b18a230b054134934ff12259656566; no voters:
I20260927 00:48:21.305049 19511 raft_consensus.cc:2781] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:21.305253 19511 raft_consensus.cc:686] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8a6cb68af646939b44618996f300b6, State: Running, Role: LEADER
I20260927 00:48:21.305589 19511 consensus_queue.cc:237] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } }
I20260927 00:48:21.308984 19037 catalog_manager.cc:5495] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8a6cb68af646939b44618996f300b6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7f8a6cb68af646939b44618996f300b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a6cb68af646939b44618996f300b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } health_report { overall_health: UNKNOWN } } }
I20260927 00:48:21.323869 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.320382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.0.98.3:44201)
0927 00:48:21.320509 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:21.320513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:21.320526 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:48:21.320641 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:21.320644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:21.320701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:21.320804 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:21.321036 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.321547 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.322439 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:21.322668 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:21.322772 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:21.323059 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:21.323455 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:21.323617 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:21.323677 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:48:21.323708 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:48:21.329330 19310 raft_consensus.cc:1261] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 1 FOLLOWER]: Refusing update from remote peer 7f8a6cb68af646939b44618996f300b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:21.329348 19179 raft_consensus.cc:1261] T c314d29452de4758bfff5d0493a21c21 P a39b62365dd041aa94433645d8dd16e1 [term 1 FOLLOWER]: Refusing update from remote peer 7f8a6cb68af646939b44618996f300b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:21.330051 19511 consensus_queue.cc:1035] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39b62365dd041aa94433645d8dd16e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:21.330235 19514 consensus_queue.cc:1035] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b18a230b054134934ff12259656566" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:21.346385 19037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:21.346546 19037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:21.348779 19037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842b8f90a7f64fe3bcb5ce171e19fd8a: Sending DeleteTablet for 3 replicas of tablet c314d29452de4758bfff5d0493a21c21
I20260927 00:48:21.349469 19421 tablet_service.cc:1514] Processing DeleteTablet for tablet c314d29452de4758bfff5d0493a21c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:21 UTC) from {username='slave'} at 127.0.0.1:53682
I20260927 00:48:21.349538 19290 tablet_service.cc:1514] Processing DeleteTablet for tablet c314d29452de4758bfff5d0493a21c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:21 UTC) from {username='slave'} at 127.0.0.1:42298
I20260927 00:48:21.350173 19526 tablet_replica.cc:330] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: stopping tablet replica
I20260927 00:48:21.350311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19097
I20260927 00:48:21.350384 19159 tablet_service.cc:1514] Processing DeleteTablet for tablet c314d29452de4758bfff5d0493a21c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:21 UTC) from {username='slave'} at 127.0.0.1:47420
I20260927 00:48:21.350487 19526 raft_consensus.cc:2229] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:21.350828 19526 raft_consensus.cc:2258] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:21.350009 19527 tablet_replica.cc:330] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: stopping tablet replica
I20260927 00:48:21.352872 19527 raft_consensus.cc:2229] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:21.353173 19527 raft_consensus.cc:2258] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:21.353500 19526 ts_tablet_manager.cc:1905] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:21.354655 19527 ts_tablet_manager.cc:1905] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:21.356334 19526 ts_tablet_manager.cc:1918] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:21.356485 19526 log.cc:1196] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/c314d29452de4758bfff5d0493a21c21
I20260927 00:48:21.356850 19526 ts_tablet_manager.cc:1939] T c314d29452de4758bfff5d0493a21c21 P 7f8a6cb68af646939b44618996f300b6: Deleting consensus metadata
I20260927 00:48:21.358016 19022 catalog_manager.cc:4841] TS 7f8a6cb68af646939b44618996f300b6 (127.0.98.3:44201): tablet c314d29452de4758bfff5d0493a21c21 (table test-table [id=85cf35a0dea44245b0c7ae345975c0c9]) successfully deleted
I20260927 00:48:21.358289 19527 ts_tablet_manager.cc:1918] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:21.358389 19527 log.cc:1196] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/c314d29452de4758bfff5d0493a21c21
I20260927 00:48:21.358711 19527 ts_tablet_manager.cc:1939] T c314d29452de4758bfff5d0493a21c21 P e9b18a230b054134934ff12259656566: Deleting consensus metadata
W20260927 00:48:21.358963 19023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45647 (error 108)
I20260927 00:48:21.359918 19021 catalog_manager.cc:4841] TS e9b18a230b054134934ff12259656566 (127.0.98.2:46393): tablet c314d29452de4758bfff5d0493a21c21 (table test-table [id=85cf35a0dea44245b0c7ae345975c0c9]) successfully deleted
I20260927 00:48:21.360045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19228
I20260927 00:48:21.359965 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:21.359672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.0.1:47436)
0927 00:48:21.359798 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:21.359865 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:48:21.362360 19023 catalog_manager.cc:4584] TS a39b62365dd041aa94433645d8dd16e1 (127.0.98.1:45647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c314d29452de4758bfff5d0493a21c21: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45647: connect: Connection refused (error 111)
I20260927 00:48:21.366720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19359
I20260927 00:48:21.372057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19006
2026-09-27T00:48:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:21 dist-test-slave-l5sb krb5kdc[19531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:21 dist-test-slave-l5sb krb5kdc[19531](info): set up 2 sockets
Sep 27 00:48:21 dist-test-slave-l5sb krb5kdc[19531](info): commencing operation
krb5kdc: starting...
W20260927 00:48:22.413697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:22 dist-test-slave-l5sb krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:22.533324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:22.641458 19544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:22.641723 19544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:22.641824 19544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:22.645722 19544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:22.645814 19544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:22.645864 19544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:22.645912 19544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:22.645951 19544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:22.651013 19544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:22.652026 19544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:22.653026 19544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:22.661528 19549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:22.661547 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:22.661528 19550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:22.661716 19544 server_base.cc:1034] running on GCE node
I20260927 00:48:22.662608 19544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:22.663602 19544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:22.664768 19544 hybrid_clock.cc:648] HybridClock initialized: now 1790470102664750 us; error 30 us; skew 500 ppm
Sep 27 00:48:22 dist-test-slave-l5sb krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:22.667475 19544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:22.668314 19544 webserver.cc:460] Webserver started at http://127.0.98.62:36631/ using document root <none> and password file <none>
I20260927 00:48:22.668848 19544 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:22.668905 19544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:22.669059 19544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:22.671103 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "66ed7d61b7954cacab58f073e49722d5"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "16fc6a91638c46e4235e1de961f1dfd1"
server_key_iv: "63458d97da2da63d191efe129d95b61a"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.671615 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "66ed7d61b7954cacab58f073e49722d5"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "16fc6a91638c46e4235e1de961f1dfd1"
server_key_iv: "63458d97da2da63d191efe129d95b61a"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.675179 19544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:22.677484 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:22.678726 19544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:48:22.678866 19544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "66ed7d61b7954cacab58f073e49722d5"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "16fc6a91638c46e4235e1de961f1dfd1"
server_key_iv: "63458d97da2da63d191efe129d95b61a"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.678975 19544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:22.693610 19544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:22.696432 19544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:22.696625 19544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:22.703514 19544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34145
I20260927 00:48:22.703562 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34145 every 8 connection(s)
I20260927 00:48:22.704440 19544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:22.707137 19612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:22.708922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19544
I20260927 00:48:22.709055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:22.709340 392 external_mini_cluster.cc:1428] Setting key 3cd640bb49a66cce097437c34bdbf5fb
I20260927 00:48:22.713487 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: Bootstrap starting.
I20260927 00:48:22.715718 19612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:22.716603 19612 log.cc:824] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:48:22 dist-test-slave-l5sb krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:22.718608 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: No bootstrap required, opened a new log
I20260927 00:48:22.722124 19612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } }
I20260927 00:48:22.722391 19612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:22.722476 19612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ed7d61b7954cacab58f073e49722d5, State: Initialized, Role: FOLLOWER
I20260927 00:48:22.722929 19612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } }
I20260927 00:48:22.723069 19612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:22.723078 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:22.710805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.0.98.62:34145)
0927 00:48:22.711232 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:22.711247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:22.711282 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:48:22.711916 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:22.711933 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:22.712242 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:22.712562 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:22.713340 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.714411 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.715262 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.715732 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.715903 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:22.718578 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:22.718583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:22.718594 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:22.718619 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:22.720293 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:22.720725 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:22.720731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:22.720735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:22.720839 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:22.721107 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:22.721112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:22.721114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:22.721472 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:22.721685 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:22.721964 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:48:22.722249 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260927 00:48:22.723167 19612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:22.723274 19612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:22.724361 19612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } }
I20260927 00:48:22.724682 19612 leader_election.cc:302] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ed7d61b7954cacab58f073e49722d5; no voters:
I20260927 00:48:22.724977 19612 leader_election.cc:288] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:22.725098 19617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:22.725365 19617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [term 1 LEADER]: Becoming Leader. State: Replica: 66ed7d61b7954cacab58f073e49722d5, State: Running, Role: LEADER
I20260927 00:48:22.725694 19617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } }
I20260927 00:48:22.726219 19612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:22.726812 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ed7d61b7954cacab58f073e49722d5. Latest consensus state: current_term: 1 leader_uuid: "66ed7d61b7954cacab58f073e49722d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } } }
I20260927 00:48:22.726928 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:22.727265 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ed7d61b7954cacab58f073e49722d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ed7d61b7954cacab58f073e49722d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34145 } } }
I20260927 00:48:22.727357 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5 [sys.catalog]: This master's current role is: LEADER
W20260927 00:48:22.730701 19632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:22.730767 19632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:22.730842 19623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:22.731513 19623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:22.736352 19623 catalog_manager.cc:1347] Generated new cluster ID: 241b164f21b34ad7b319fba6ba92c536
I20260927 00:48:22.736416 19623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:22.752393 19623 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:22.753260 19623 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:22.767195 19623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: Generated new TSK 0
I20260927 00:48:22.767890 19623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:48:22.809257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:22.918001 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:22.918241 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:22.918280 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:22.922106 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:22.922152 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:22.922206 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:22.927440 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:22.928426 19638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:22.929394 19638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:22.939044 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:22.939044 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:22.939123 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:22.939092 19638 server_base.cc:1034] running on GCE node
I20260927 00:48:22.939683 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:22.940307 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:22.941493 19638 hybrid_clock.cc:648] HybridClock initialized: now 1790470102941478 us; error 28 us; skew 500 ppm
Sep 27 00:48:22 dist-test-slave-l5sb krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:22.944317 19638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:48:22.945484 19638 webserver.cc:460] Webserver started at http://127.0.98.1:32843/ using document root <none> and password file <none>
I20260927 00:48:22.946120 19638 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:22.946205 19638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:22.946409 19638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:22.948391 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "812c4b94506f34ab20dfb3af4348abe5"
server_key_iv: "cd0842787ddc8fb3efa115abd420a8e1"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.948948 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "812c4b94506f34ab20dfb3af4348abe5"
server_key_iv: "cd0842787ddc8fb3efa115abd420a8e1"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.952533 19638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 00:48:22.954833 19653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:22.955982 19638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:22.956177 19638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc"
format_stamp: "Formatted at 2026-09-27 00:48:22 on dist-test-slave-l5sb"
server_key: "812c4b94506f34ab20dfb3af4348abe5"
server_key_iv: "cd0842787ddc8fb3efa115abd420a8e1"
server_key_version: "encryptionkey@0"
I20260927 00:48:22.956281 19638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:22.975662 19638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:22.978681 19638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:22.978878 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:22.979480 19638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:22.980288 19638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:22.980353 19638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:22.980423 19638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:22.980468 19638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:22.989652 19638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43895
I20260927 00:48:22.989687 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43895 every 8 connection(s)
I20260927 00:48:22.990581 19638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:22.995209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19638
I20260927 00:48:22.995349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:22.995632 392 external_mini_cluster.cc:1428] Setting key ab0661be7a451e810af59985696281cf
Sep 27 00:48:22 dist-test-slave-l5sb krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:23.004243 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:22.992287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41821 (local address 127.0.98.62:34145)
0927 00:48:22.992427 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:22.992432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:22.993572 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 00:48:22.994510 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:22.994513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:22.994565 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:22.994647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:22.995980 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.996669 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.997909 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.998152 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.998259 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.000757 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:23.000766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:23.000768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:23.000790 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:23.002262 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.002765 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.002771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.002772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.002850 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.003178 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.003182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.003184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.003367 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:23.003470 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.003941 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.004064 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:48:23.005036 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:22.992733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34145 (local address 127.0.98.1:41821)
0927 00:48:22.993414 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:22.993457 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:22.994327 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:22.994826 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:22.994833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:22.995203 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:22.995832 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:22.995848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.996811 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.996815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:22.997745 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:22.997754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:22.998311 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:22.998315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:22.998483 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:22.999127 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:22.999143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:23.000598 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:23.002391 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.002396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.002412 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.002667 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.002937 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.002941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.002943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.003078 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.003468 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:23.003475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:23.003719 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.003899 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.004227 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":113,"threads_started":1}
I20260927 00:48:23.006271 19767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34145
I20260927 00:48:23.006515 19767 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:48:23.007047 19767 heartbeater.cc:511] Master 127.0.98.62:34145 requested a full tablet report, sending...
I20260927 00:48:23.008736 19576 ts_manager.cc:194] Registered new tserver with Master: 7e7d82dc30c446e09dbd3a439e3eb7fc (127.0.98.1:43895)
I20260927 00:48:23.009940 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:48:23.024379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:23.143540 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:23.143790 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:23.143851 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:23.147951 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:23.148006 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:23.148101 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:23.153282 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:23.154368 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:23.155355 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:23.164714 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:23.164871 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:23.164866 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:23.164960 19772 server_base.cc:1034] running on GCE node
I20260927 00:48:23.165346 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:23.166090 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:23.167265 19772 hybrid_clock.cc:648] HybridClock initialized: now 1790470103167245 us; error 32 us; skew 500 ppm
Sep 27 00:48:23 dist-test-slave-l5sb krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:23.169968 19772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:48:23.170738 19772 webserver.cc:460] Webserver started at http://127.0.98.2:39301/ using document root <none> and password file <none>
I20260927 00:48:23.171258 19772 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:23.171316 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:23.171468 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:23.173336 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "327adbef9dcf451491c20a491b862850"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "942e1c5e9dbd796dd4038b28cf3bb280"
server_key_iv: "58c3fc033b5bdb27bc531409059322dc"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.173894 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "327adbef9dcf451491c20a491b862850"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "942e1c5e9dbd796dd4038b28cf3bb280"
server_key_iv: "58c3fc033b5bdb27bc531409059322dc"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.177563 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:48:23.179960 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.181131 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:48:23.181242 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "327adbef9dcf451491c20a491b862850"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "942e1c5e9dbd796dd4038b28cf3bb280"
server_key_iv: "58c3fc033b5bdb27bc531409059322dc"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.181340 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:23.201100 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:23.204478 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:23.204686 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:23.205271 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:23.206146 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:23.206192 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.206255 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:23.206287 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.215164 19772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44723
I20260927 00:48:23.215188 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44723 every 8 connection(s)
I20260927 00:48:23.216078 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:23.220539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19772
I20260927 00:48:23.220677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:23.220947 392 external_mini_cluster.cc:1428] Setting key be043674b7975347fe29a102e51198aa
Sep 27 00:48:23 dist-test-slave-l5sb krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:48:23.230827 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.217882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39679 (local address 127.0.98.62:34145)
0927 00:48:23.218027 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.218031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.218859 (+ 828us) server_negotiation.cc:407] Connection header received
0927 00:48:23.219801 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.219805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.219861 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.219934 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:23.221176 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.221907 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.223165 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.223402 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.223519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.226355 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:23.226359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:23.226362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:23.226390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:23.228277 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.228803 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.228812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.228814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.228965 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.229505 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.229508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.229510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.229717 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:23.229936 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.230515 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.230646 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:48:23.231675 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.218275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34145 (local address 127.0.98.2:39679)
0927 00:48:23.218719 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.218762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.219591 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.220112 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.220119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.220452 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:23.221033 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.221056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.222033 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.222037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.223036 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.223046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.223544 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.223549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.223725 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.224305 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:23.224321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:23.226120 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:23.228408 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.228414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.228429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.228694 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.229152 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.229155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.229157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.229384 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.229933 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:23.229939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:23.230261 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.230458 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.230777 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260927 00:48:23.232867 19901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:48:23.233134 19901 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:23.233712 19901 heartbeater.cc:511] Master 127.0.98.62:34145 requested a full tablet report, sending...
I20260927 00:48:23.234949 19576 ts_manager.cc:194] Registered new tserver with Master: 327adbef9dcf451491c20a491b862850 (127.0.98.2:44723)
I20260927 00:48:23.235620 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:48:23.249683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:23.361873 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:23.362138 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:23.362229 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:23.366266 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:23.366331 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:23.366411 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:23.371606 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:23.372591 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:23.373561 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:23.382916 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:23.382936 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:23.383121 19906 server_base.cc:1034] running on GCE node
W20260927 00:48:23.382920 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:23.383680 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:23.384305 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:23.385490 19906 hybrid_clock.cc:648] HybridClock initialized: now 1790470103385469 us; error 34 us; skew 500 ppm
Sep 27 00:48:23 dist-test-slave-l5sb krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:23.388190 19906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:48:23.388986 19906 webserver.cc:460] Webserver started at http://127.0.98.3:44053/ using document root <none> and password file <none>
I20260927 00:48:23.389480 19906 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:23.389544 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:23.389701 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:23.391647 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e8d4e3187044b92b12704213df6cb61"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "e6f73b970d43a896e20775761c4ce46c"
server_key_iv: "530cdb785c85199891591b294d38ef77"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.392138 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e8d4e3187044b92b12704213df6cb61"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "e6f73b970d43a896e20775761c4ce46c"
server_key_iv: "530cdb785c85199891591b294d38ef77"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.395673 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:48:23.397914 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.399056 19906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:23.399173 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "8e8d4e3187044b92b12704213df6cb61"
format_stamp: "Formatted at 2026-09-27 00:48:23 on dist-test-slave-l5sb"
server_key: "e6f73b970d43a896e20775761c4ce46c"
server_key_iv: "530cdb785c85199891591b294d38ef77"
server_key_version: "encryptionkey@0"
I20260927 00:48:23.399268 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:23.426802 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:23.430124 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:23.430321 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:23.430898 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:23.431694 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:23.431737 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.431798 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:23.431831 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:23.440697 19906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44535
I20260927 00:48:23.440730 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44535 every 8 connection(s)
I20260927 00:48:23.441541 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:23.445760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 19906
I20260927 00:48:23.445952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:23.446234 392 external_mini_cluster.cc:1428] Setting key ccdd11bd276982bcc82d5f5c3666ce46
Sep 27 00:48:23 dist-test-slave-l5sb krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:23.454926 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.443308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60223 (local address 127.0.98.62:34145)
0927 00:48:23.443442 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.443446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.444292 (+ 846us) server_negotiation.cc:407] Connection header received
0927 00:48:23.445191 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.445195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.445250 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.445353 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:23.446696 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.447375 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.448729 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.448956 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.449065 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.451485 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:23.451488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:23.451490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:23.451514 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:23.452903 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.453347 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.453350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.453352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.453425 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.453728 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.453731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.453732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.453948 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:23.454025 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.454611 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.454735 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:48:23.455729 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.443675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34145 (local address 127.0.98.3:60223)
0927 00:48:23.444152 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.444192 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.444975 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.445523 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.445531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.445957 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:23.446522 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.446544 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.447497 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.447501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.448607 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.448617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.449104 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.449108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.449281 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.449826 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:23.449849 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:23.451338 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:23.453001 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.453005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.453017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.453259 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.453511 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:23.453513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:23.453515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:23.453652 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:23.454067 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:23.454072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:23.454298 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.454553 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.454866 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260927 00:48:23.456820 20035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34145
I20260927 00:48:23.457098 20035 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:23.457571 20035 heartbeater.cc:511] Master 127.0.98.62:34145 requested a full tablet report, sending...
I20260927 00:48:23.458626 19576 ts_manager.cc:194] Registered new tserver with Master: 8e8d4e3187044b92b12704213df6cb61 (127.0.98.3:44535)
I20260927 00:48:23.459374 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60223
I20260927 00:48:23.459996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:23.469434 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.462207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.62:34145)
0927 00:48:23.462323 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.462329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.462550 (+ 221us) server_negotiation.cc:407] Connection header received
0927 00:48:23.462688 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.462691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.462734 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.462819 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:23.463495 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.463978 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.464808 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.465025 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.465127 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.466412 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:23.466416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:23.466419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:23.466446 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:23.467779 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.468241 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.468245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.468246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.468325 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:23.468597 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:23.468600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:23.468602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:23.468804 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:23.468940 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.469167 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.469278 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:48:23.472052 19575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:23.474993 19575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:23.489521 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.483640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.98.2:44723)
0927 00:48:23.484399 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.484442 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.484472 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:48:23.484567 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.484575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.484789 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.484999 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:23.485298 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.486496 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.488018 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.488740 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.488824 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.488945 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.489250 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.489358 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":89,"threads_started":1}
I20260927 00:48:23.489487 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.483420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44723 (local address 127.0.0.1:53128)
0927 00:48:23.483865 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.483911 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.484040 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.484974 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.484980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.485014 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:23.485148 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.485154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.486633 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.486637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.487901 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.487909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.488861 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.488865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.488978 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.489074 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.489205 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.489330 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260927 00:48:23.490123 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.484223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44535 (local address 127.0.0.1:43132)
0927 00:48:23.484545 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.484558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.484634 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.485453 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.485456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.485473 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:23.485593 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.485600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.487035 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.487040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.488291 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.488301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.489422 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.489425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.489515 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.489612 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.489847 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.489924 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260927 00:48:23.490134 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.484399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.0.98.3:44535)
0927 00:48:23.484936 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.484952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.484972 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:48:23.485042 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.485047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.485246 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.485724 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:23.485921 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.486902 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.488433 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.489138 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.489235 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.489361 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.489884 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.489986 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":146,"threads_started":1}
I20260927 00:48:23.491307 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.483595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43895 (local address 127.0.0.1:43894)
0927 00:48:23.484005 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.484022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.484141 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.485819 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.485822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.485840 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:23.485945 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.485951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.487150 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.487153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.489225 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.489234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.490895 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.490899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.491028 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.491051 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.491102 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.491169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1015,"thread_start_us":102,"threads_started":1}
I20260927 00:48:23.491899 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.484545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.1:43895)
0927 00:48:23.485301 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.485317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.485339 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:48:23.485413 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.485418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.485672 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.485844 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:23.486138 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.487022 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.489357 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.490757 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.491571 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.491683 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.491722 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.491788 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":74,"threads_started":1}
I20260927 00:48:23.492858 19969 tablet_service.cc:1467] Processing CreateTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 (DEFAULT_TABLE table=test-table [id=5963acc53d214993b1015103c50a7245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:23.492852 19835 tablet_service.cc:1467] Processing CreateTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 (DEFAULT_TABLE table=test-table [id=5963acc53d214993b1015103c50a7245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:23.494037 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b47e31b88514055ae350cbdffb48bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:23.494037 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b47e31b88514055ae350cbdffb48bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:23.494434 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 (DEFAULT_TABLE table=test-table [id=5963acc53d214993b1015103c50a7245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:23.495430 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b47e31b88514055ae350cbdffb48bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:23.501053 20052 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Bootstrap starting.
I20260927 00:48:23.501679 20053 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Bootstrap starting.
I20260927 00:48:23.501959 20054 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Bootstrap starting.
I20260927 00:48:23.503897 20054 tablet_bootstrap.cc:654] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:23.503940 20052 tablet_bootstrap.cc:654] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:23.504366 20053 tablet_bootstrap.cc:654] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:23.505095 20054 log.cc:824] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:23.505198 20053 log.cc:824] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:23.505810 20052 log.cc:824] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:23.507023 20054 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: No bootstrap required, opened a new log
I20260927 00:48:23.507093 20053 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: No bootstrap required, opened a new log
I20260927 00:48:23.507227 20054 ts_tablet_manager.cc:1397] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:48:23.507274 20053 ts_tablet_manager.cc:1397] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:48:23.507529 20052 tablet_bootstrap.cc:492] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: No bootstrap required, opened a new log
I20260927 00:48:23.507688 20052 ts_tablet_manager.cc:1397] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:48:23.510147 20054 raft_consensus.cc:348] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.510391 20054 raft_consensus.cc:374] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:23.510455 20054 raft_consensus.cc:729] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8d4e3187044b92b12704213df6cb61, State: Initialized, Role: FOLLOWER
I20260927 00:48:23.510895 20054 consensus_queue.cc:260] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.511029 20053 raft_consensus.cc:348] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.511291 20053 raft_consensus.cc:374] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:23.511353 20053 raft_consensus.cc:729] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327adbef9dcf451491c20a491b862850, State: Initialized, Role: FOLLOWER
I20260927 00:48:23.511324 20052 raft_consensus.cc:348] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.511550 20052 raft_consensus.cc:374] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:23.511598 20052 raft_consensus.cc:729] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7d82dc30c446e09dbd3a439e3eb7fc, State: Initialized, Role: FOLLOWER
I20260927 00:48:23.511792 20053 consensus_queue.cc:260] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.512003 20035 heartbeater.cc:503] Master 127.0.98.62:34145 was elected leader, sending a full tablet report...
I20260927 00:48:23.512377 20052 consensus_queue.cc:260] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.512720 20053 ts_tablet_manager.cc:1428] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:48:23.513022 19901 heartbeater.cc:503] Master 127.0.98.62:34145 was elected leader, sending a full tablet report...
I20260927 00:48:23.513047 19767 heartbeater.cc:503] Master 127.0.98.62:34145 was elected leader, sending a full tablet report...
I20260927 00:48:23.513290 20052 ts_tablet_manager.cc:1428] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:48:23.513584 20054 ts_tablet_manager.cc:1428] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 00:48:23.692487 20036 tablet.cc:2369] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:23.717271 19902 tablet.cc:2369] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:23.742475 19768 tablet.cc:2369] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:23.954406 20058 raft_consensus.cc:482] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:23.954607 20058 raft_consensus.cc:504] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.955694 20058 leader_election.cc:288] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 327adbef9dcf451491c20a491b862850 (127.0.98.2:44723), 7e7d82dc30c446e09dbd3a439e3eb7fc (127.0.98.1:43895)
I20260927 00:48:23.961225 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.956244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44159 (local address 127.0.98.1:43895)
0927 00:48:23.956356 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.956360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.956448 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:48:23.956540 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.956543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.956598 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.956685 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:23.957161 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.957927 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.959949 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.960583 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.960685 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.960773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.960988 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.961078 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:48:23.961320 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.956049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43895 (local address 127.0.98.3:44159)
0927 00:48:23.956351 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.956364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.956461 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.956702 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.956705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.956750 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:23.957022 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.957033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.958079 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.958085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.959823 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.959832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.960697 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.960701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.960856 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.960882 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.960951 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.961016 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":225,"spinlock_wait_cycles":4736,"thread_start_us":71,"threads_started":1}
I20260927 00:48:23.961367 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.956091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57059 (local address 127.0.98.2:44723)
0927 00:48:23.956236 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:23.956240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:23.956385 (+ 145us) server_negotiation.cc:407] Connection header received
0927 00:48:23.956466 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:23.956472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:23.956527 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:23.956672 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:23.957098 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.957927 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.959950 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.960568 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.960689 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.960753 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:23.961139 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 00:48:23.961240 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:23.961328 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:23.956004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44723 (local address 127.0.98.3:57059)
0927 00:48:23.956206 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:23.956223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:23.956336 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:23.956702 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:23.956705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:23.956744 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:23.956970 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.956977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.958115 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.958120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.959820 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:23.959832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:23.960819 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:23.960823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:23.960968 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:23.960991 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:48:23.961069 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:48:23.961139 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":199,"spinlock_wait_cycles":28800}
I20260927 00:48:23.962100 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b47e31b88514055ae350cbdffb48bb1" candidate_uuid: "8e8d4e3187044b92b12704213df6cb61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" is_pre_election: true
I20260927 00:48:23.962152 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b47e31b88514055ae350cbdffb48bb1" candidate_uuid: "8e8d4e3187044b92b12704213df6cb61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327adbef9dcf451491c20a491b862850" is_pre_election: true
I20260927 00:48:23.962438 19855 raft_consensus.cc:2454] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e8d4e3187044b92b12704213df6cb61 in term 0.
I20260927 00:48:23.962451 19721 raft_consensus.cc:2454] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e8d4e3187044b92b12704213df6cb61 in term 0.
I20260927 00:48:23.962934 19925 leader_election.cc:302] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7d82dc30c446e09dbd3a439e3eb7fc, 8e8d4e3187044b92b12704213df6cb61; no voters:
I20260927 00:48:23.963236 20058 raft_consensus.cc:2781] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:23.963338 20058 raft_consensus.cc:482] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:23.963397 20058 raft_consensus.cc:3037] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:23.964470 20058 raft_consensus.cc:504] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.964886 20058 leader_election.cc:288] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [CANDIDATE]: Term 1 election: Requested vote from peers 327adbef9dcf451491c20a491b862850 (127.0.98.2:44723), 7e7d82dc30c446e09dbd3a439e3eb7fc (127.0.98.1:43895)
I20260927 00:48:23.965209 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b47e31b88514055ae350cbdffb48bb1" candidate_uuid: "8e8d4e3187044b92b12704213df6cb61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327adbef9dcf451491c20a491b862850"
I20260927 00:48:23.965234 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b47e31b88514055ae350cbdffb48bb1" candidate_uuid: "8e8d4e3187044b92b12704213df6cb61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc"
I20260927 00:48:23.965334 19855 raft_consensus.cc:3037] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:23.965355 19721 raft_consensus.cc:3037] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:23.966387 19855 raft_consensus.cc:2454] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e8d4e3187044b92b12704213df6cb61 in term 1.
I20260927 00:48:23.966461 19721 raft_consensus.cc:2454] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e8d4e3187044b92b12704213df6cb61 in term 1.
I20260927 00:48:23.966702 19924 leader_election.cc:302] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327adbef9dcf451491c20a491b862850, 8e8d4e3187044b92b12704213df6cb61; no voters:
I20260927 00:48:23.966931 20058 raft_consensus.cc:2781] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:23.967165 20058 raft_consensus.cc:686] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 1 LEADER]: Becoming Leader. State: Replica: 8e8d4e3187044b92b12704213df6cb61, State: Running, Role: LEADER
I20260927 00:48:23.967521 20058 consensus_queue.cc:237] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } }
I20260927 00:48:23.970837 19576 catalog_manager.cc:5495] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e8d4e3187044b92b12704213df6cb61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8e8d4e3187044b92b12704213df6cb61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e8d4e3187044b92b12704213df6cb61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44535 } health_report { overall_health: HEALTHY } } }
I20260927 00:48:24.049075 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.045431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.0.98.3:44535)
0927 00:48:24.045655 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:24.045658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:24.045673 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:48:24.045879 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:24.045881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:24.045940 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:24.046096 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:24.046336 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.046846 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.047681 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.047900 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.047979 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.048250 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:24.048685 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:24.048777 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:24.048821 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 00:48:24.048867 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260927 00:48:24.054945 19855 raft_consensus.cc:1261] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 1 FOLLOWER]: Refusing update from remote peer 8e8d4e3187044b92b12704213df6cb61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:24.054945 19721 raft_consensus.cc:1261] T 6b47e31b88514055ae350cbdffb48bb1 P 7e7d82dc30c446e09dbd3a439e3eb7fc [term 1 FOLLOWER]: Refusing update from remote peer 8e8d4e3187044b92b12704213df6cb61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:24.055624 20058 consensus_queue.cc:1035] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7d82dc30c446e09dbd3a439e3eb7fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:24.055821 20062 consensus_queue.cc:1035] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327adbef9dcf451491c20a491b862850" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:24.071894 19576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:24.072057 19576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:24.074640 19576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ed7d61b7954cacab58f073e49722d5: Sending DeleteTablet for 3 replicas of tablet 6b47e31b88514055ae350cbdffb48bb1
I20260927 00:48:24.075439 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53128
I20260927 00:48:24.075584 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43132
I20260927 00:48:24.075569 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b47e31b88514055ae350cbdffb48bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43894
I20260927 00:48:24.076197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19638
I20260927 00:48:24.077587 20080 tablet_replica.cc:330] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: stopping tablet replica
I20260927 00:48:24.077944 20080 raft_consensus.cc:2229] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:24.078382 20080 raft_consensus.cc:2258] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:24.078651 20079 tablet_replica.cc:330] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850: stopping tablet replica
I20260927 00:48:24.078975 20079 raft_consensus.cc:2229] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:24.079286 20079 raft_consensus.cc:2258] T 6b47e31b88514055ae350cbdffb48bb1 P 327adbef9dcf451491c20a491b862850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:24.080400 20080 ts_tablet_manager.cc:1905] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:24.082996 20080 ts_tablet_manager.cc:1918] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:24.083107 20080 log.cc:1196] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/6b47e31b88514055ae350cbdffb48bb1
I20260927 00:48:24.083482 20080 ts_tablet_manager.cc:1939] T 6b47e31b88514055ae350cbdffb48bb1 P 8e8d4e3187044b92b12704213df6cb61: Deleting consensus metadata
I20260927 00:48:24.084092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19772
W20260927 00:48:24.084085 19563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43895 (error 108)
I20260927 00:48:24.086319 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.085821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43895 (local address 127.0.0.1:43900)
0927 00:48:24.086098 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:24.086201 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":55,"threads_started":1}
I20260927 00:48:24.086300 19563 catalog_manager.cc:4841] TS 8e8d4e3187044b92b12704213df6cb61 (127.0.98.3:44535): tablet 6b47e31b88514055ae350cbdffb48bb1 (table test-table [id=5963acc53d214993b1015103c50a7245]) successfully deleted
W20260927 00:48:24.086781 19563 catalog_manager.cc:4584] TS 7e7d82dc30c446e09dbd3a439e3eb7fc (127.0.98.1:43895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b47e31b88514055ae350cbdffb48bb1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43895: connect: Connection refused (error 111)
I20260927 00:48:24.091020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19906
I20260927 00:48:24.092597 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.091988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44723 (local address 127.0.0.1:53142)
0927 00:48:24.092441 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:24.092491 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
I20260927 00:48:24.095958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 19544
2026-09-27T00:48:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2726 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:24Z Disabled control of system clock
I20260927 00:48:24.126295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43369
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:24.234136 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:24.234402 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:24.234436 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:24.238489 20087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:24.238548 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:24.238564 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:24.238581 20087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:24.238596 20087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:24.243263 20087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:24.244272 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:24.245577 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:24.253986 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.253979 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.253979 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.254403 20087 server_base.cc:1034] running on GCE node
I20260927 00:48:24.255117 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:24.256232 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:24.257423 20087 hybrid_clock.cc:648] HybridClock initialized: now 1790470104257410 us; error 30 us; skew 500 ppm
I20260927 00:48:24.259248 20087 webserver.cc:460] Webserver started at http://127.0.98.62:43629/ using document root <none> and password file <none>
I20260927 00:48:24.259805 20087 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:24.259892 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:24.260108 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:24.262149 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "fc784f8949b841a999dea7fefd93ab8f"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "c1428f478d9bcede377fb9f18f0d4626"
server_key_iv: "1a81ac2952cf4ca88764abd5cb09c83d"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.262686 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc784f8949b841a999dea7fefd93ab8f"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "c1428f478d9bcede377fb9f18f0d4626"
server_key_iv: "1a81ac2952cf4ca88764abd5cb09c83d"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.266140 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:24.268285 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.269433 20087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:24.269559 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "fc784f8949b841a999dea7fefd93ab8f"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "c1428f478d9bcede377fb9f18f0d4626"
server_key_iv: "1a81ac2952cf4ca88764abd5cb09c83d"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.269670 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:24.304581 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:24.305349 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:24.305550 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:24.312594 20087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43369
I20260927 00:48:24.312638 20153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43369 every 8 connection(s)
I20260927 00:48:24.313544 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:24.317199 20154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:24.321935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20087
I20260927 00:48:24.322070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:24.322410 392 external_mini_cluster.cc:1428] Setting key eb68a56da7b1e4f41d5593dba5276c0c
I20260927 00:48:24.323639 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: Bootstrap starting.
I20260927 00:48:24.326107 20154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:24.326877 20154 log.cc:824] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:24.328764 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: No bootstrap required, opened a new log
I20260927 00:48:24.331723 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.323961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.98.62:43369)
0927 00:48:24.324641 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:24.324656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:24.324690 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:48:24.325310 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:24.325327 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:24.325716 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:24.326102 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:24.326563 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.327627 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.328484 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.329028 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.329188 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.330075 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:24.330085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:24.330097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:24.330464 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:24.330519 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:24.330539 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:24.330642 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:24.330757 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:48:24.331015 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260927 00:48:24.332726 20154 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } }
I20260927 00:48:24.333036 20154 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:24.333102 20154 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc784f8949b841a999dea7fefd93ab8f, State: Initialized, Role: FOLLOWER
I20260927 00:48:24.333581 20154 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } }
I20260927 00:48:24.333746 20154 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:24.333839 20154 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:24.333984 20154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:24.334952 20154 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } }
I20260927 00:48:24.335306 20154 leader_election.cc:302] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc784f8949b841a999dea7fefd93ab8f; no voters:
I20260927 00:48:24.335577 20154 leader_election.cc:288] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:24.335724 20159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:24.335986 20159 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [term 1 LEADER]: Becoming Leader. State: Replica: fc784f8949b841a999dea7fefd93ab8f, State: Running, Role: LEADER
I20260927 00:48:24.336318 20159 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } }
I20260927 00:48:24.337002 20154 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:24.338145 20161 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc784f8949b841a999dea7fefd93ab8f. Latest consensus state: current_term: 1 leader_uuid: "fc784f8949b841a999dea7fefd93ab8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } } }
I20260927 00:48:24.338399 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc784f8949b841a999dea7fefd93ab8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc784f8949b841a999dea7fefd93ab8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43369 } } }
I20260927 00:48:24.338527 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:24.338575 20161 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:24.339334 20168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:24.342554 20168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:24.348254 20168 catalog_manager.cc:1347] Generated new cluster ID: c2bc9356fbd14bf9ab5220818210ef9f
I20260927 00:48:24.348320 20168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:24.362500 20168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:24.363734 20168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:24.388659 20168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: Generated new TSK 0
I20260927 00:48:24.389391 20168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:24.394069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--builtin_ntp_servers=127.0.98.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:48:24.521945 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:24.522203 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:24.522243 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:24.526726 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:24.526780 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:24.526836 20178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:24.531692 20178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:24.532766 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:24.534216 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:24.544796 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.544757 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.545032 20178 server_base.cc:1034] running on GCE node
W20260927 00:48:24.544757 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.545554 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:24.546393 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:24.547616 20178 hybrid_clock.cc:648] HybridClock initialized: now 1790470104547590 us; error 38 us; skew 500 ppm
I20260927 00:48:24.549930 20178 webserver.cc:460] Webserver started at http://127.0.98.1:44449/ using document root <none> and password file <none>
I20260927 00:48:24.550588 20178 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:24.550693 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:24.550954 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:24.553244 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff458041ca04441fa739f677fdd5de7c"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "783071b99855dabbf9d8000bdba8abaa"
server_key_iv: "8e0220d50abef3db7a4a368eb85fa59b"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.553934 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff458041ca04441fa739f677fdd5de7c"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "783071b99855dabbf9d8000bdba8abaa"
server_key_iv: "8e0220d50abef3db7a4a368eb85fa59b"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.558074 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:48:24.560647 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.562075 20178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:48:24.562206 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal
uuid: "ff458041ca04441fa739f677fdd5de7c"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "783071b99855dabbf9d8000bdba8abaa"
server_key_iv: "8e0220d50abef3db7a4a368eb85fa59b"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.562316 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:24.618119 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:24.619131 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:24.619336 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:24.619969 20178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:24.620965 20178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:24.621042 20178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.621109 20178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:24.621140 20178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.630872 20178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37065
I20260927 00:48:24.630901 20305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37065 every 8 connection(s)
I20260927 00:48:24.631795 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:24.641996 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.633881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43241 (local address 127.0.98.62:43369)
0927 00:48:24.634030 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:24.634036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:24.635174 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 00:48:24.636726 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:24.636730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:24.636795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:24.636876 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:24.638028 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.638745 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.639966 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.640191 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.640303 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.641451 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:24.641455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:24.641458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:24.641477 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:24.641495 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:24.641502 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:24.641590 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:24.641753 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:48:24.641849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:24.642031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20178
I20260927 00:48:24.642172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:24.642441 392 external_mini_cluster.cc:1428] Setting key 521a5b93b27ff091d3f22a21f1828180
I20260927 00:48:24.642982 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.634290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43369 (local address 127.0.98.1:43241)
0927 00:48:24.634972 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:24.635017 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:24.636482 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:24.637189 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:24.637197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:24.637287 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:24.637875 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:24.637892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.638887 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.638891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:24.639822 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:24.639831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.640351 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.640355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:24.640519 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.641187 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:24.641189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:24.641230 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:24.641292 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:24.641592 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:24.641603 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:24.641613 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:48:24.641804 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 00:48:24.642103 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260927 00:48:24.644368 20306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43369
I20260927 00:48:24.644490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--builtin_ntp_servers=127.0.98.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:24.644704 20306 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:24.645287 20306 heartbeater.cc:511] Master 127.0.98.62:43369 requested a full tablet report, sending...
I20260927 00:48:24.647006 20118 ts_manager.cc:194] Registered new tserver with Master: ff458041ca04441fa739f677fdd5de7c (127.0.98.1:37065)
I20260927 00:48:24.648311 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43241
W20260927 00:48:24.755811 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:24.756108 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:24.756151 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:24.760342 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:24.760401 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:24.760486 20309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:24.765497 20309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:24.766685 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:24.768149 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:24.780507 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.780500 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.780675 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.780862 20309 server_base.cc:1034] running on GCE node
I20260927 00:48:24.781301 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:24.782056 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:24.783234 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790470104783194 us; error 53 us; skew 500 ppm
I20260927 00:48:24.785240 20309 webserver.cc:460] Webserver started at http://127.0.98.2:35551/ using document root <none> and password file <none>
I20260927 00:48:24.785904 20309 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:24.785971 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:24.786232 20309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:24.788319 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffaabc33c9ed4066beddf05055b82f17"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "6766aff3bae96c127f40e6aa5e6774d9"
server_key_iv: "787bb3bc36f5b02565070344f0d4d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.788878 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffaabc33c9ed4066beddf05055b82f17"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "6766aff3bae96c127f40e6aa5e6774d9"
server_key_iv: "787bb3bc36f5b02565070344f0d4d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.792676 20309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:48:24.795183 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.796432 20309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:24.796535 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal
uuid: "ffaabc33c9ed4066beddf05055b82f17"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "6766aff3bae96c127f40e6aa5e6774d9"
server_key_iv: "787bb3bc36f5b02565070344f0d4d0cf"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.796682 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:24.820048 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:24.820765 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:24.820966 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:24.821563 20309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:24.822563 20309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:24.822619 20309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.822691 20309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:24.822722 20309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.832115 20309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39681
I20260927 00:48:24.832126 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39681 every 8 connection(s)
I20260927 00:48:24.833052 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:24.842540 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.834850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34003 (local address 127.0.98.62:43369)
0927 00:48:24.834981 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:24.834984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:24.836029 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 00:48:24.837507 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:24.837510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:24.837569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:24.837677 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:24.838778 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.839397 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.840543 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.840771 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.840881 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.842005 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:24.842009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:24.842011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:24.842024 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:24.842052 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:24.842060 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:24.842179 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:24.842335 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 00:48:24.842392 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:48:24.843007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20309
I20260927 00:48:24.843116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:24.843401 392 external_mini_cluster.cc:1428] Setting key 4d4c85d990c34638556acc80744d5ef3
I20260927 00:48:24.843580 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:24.835254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43369 (local address 127.0.98.2:34003)
0927 00:48:24.835871 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:24.835913 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:24.837299 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:24.837879 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:24.837886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:24.837969 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:24.838612 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:24.838634 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.839520 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.839523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:24.840417 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:24.840426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:24.840919 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:24.840923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:24.841093 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:24.841705 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:24.841706 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:24.841745 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:24.841862 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:24.842180 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:24.842191 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:24.842200 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:48:24.842335 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 00:48:24.842599 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260927 00:48:24.844688 20437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43369
I20260927 00:48:24.844913 20437 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:24.845417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--builtin_ntp_servers=127.0.98.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:48:24.845424 20437 heartbeater.cc:511] Master 127.0.98.62:43369 requested a full tablet report, sending...
I20260927 00:48:24.846539 20117 ts_manager.cc:194] Registered new tserver with Master: ffaabc33c9ed4066beddf05055b82f17 (127.0.98.2:39681)
I20260927 00:48:24.847174 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34003
W20260927 00:48:24.958364 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:24.958652 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:24.958722 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:24.962843 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:24.962920 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:24.963004 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:24.967828 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:24.968931 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:24.970372 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:24.980269 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.980355 20440 server_base.cc:1034] running on GCE node
W20260927 00:48:24.980293 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:24.980290 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:24.981055 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:24.981695 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:24.982887 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790470104982849 us; error 49 us; skew 500 ppm
I20260927 00:48:24.984830 20440 webserver.cc:460] Webserver started at http://127.0.98.3:40819/ using document root <none> and password file <none>
I20260927 00:48:24.985414 20440 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:24.985500 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:24.985702 20440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:24.987826 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data/instance:
uuid: "adfd7d36e15848458663e66414b45bbb"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "5f5ebc87ced8730bd4b74746046245d2"
server_key_iv: "af02b33da9dabaf6a51eabb07f1279fc"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.988343 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal/instance:
uuid: "adfd7d36e15848458663e66414b45bbb"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "5f5ebc87ced8730bd4b74746046245d2"
server_key_iv: "af02b33da9dabaf6a51eabb07f1279fc"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.992033 20440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:48:24.994410 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:24.995661 20440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:24.995792 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal
uuid: "adfd7d36e15848458663e66414b45bbb"
format_stamp: "Formatted at 2026-09-27 00:48:24 on dist-test-slave-l5sb"
server_key: "5f5ebc87ced8730bd4b74746046245d2"
server_key_iv: "af02b33da9dabaf6a51eabb07f1279fc"
server_key_version: "encryptionkey@0"
I20260927 00:48:24.995904 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:25.006850 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:25.007548 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:25.007761 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:25.008368 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:25.009320 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:25.009393 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:25.009467 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:25.009513 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:25.018234 20440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41431
I20260927 00:48:25.018350 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41431 every 8 connection(s)
I20260927 00:48:25.019232 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:25.021842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20440
I20260927 00:48:25.021991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:25.022329 392 external_mini_cluster.cc:1428] Setting key 757496ade4f25921fe9d6d6c2e486ff8
I20260927 00:48:25.028829 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.021110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41971 (local address 127.0.98.62:43369)
0927 00:48:25.021250 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.021254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.022508 (+ 1254us) server_negotiation.cc:407] Connection header received
0927 00:48:25.024167 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.024170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.024223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.024315 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:25.025252 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.025831 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.027057 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.027282 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.027357 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.028417 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:25.028419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:25.028422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:25.028433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:25.028453 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:25.028460 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:25.028522 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.028667 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.028718 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:25.029753 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.021497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43369 (local address 127.0.98.3:41971)
0927 00:48:25.022341 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.022404 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.023958 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.024447 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.024479 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.024602 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:25.025125 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.025139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.025982 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.025985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.026954 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.026963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.027414 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.027417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.027573 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.028168 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:48:25.028169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:25.028211 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:48:25.028275 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:25.028540 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:25.028547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:25.028551 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.028654 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.028886 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":110,"threads_started":1}
I20260927 00:48:25.030944 20568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43369
I20260927 00:48:25.031167 20568 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:25.031651 20568 heartbeater.cc:511] Master 127.0.98.62:43369 requested a full tablet report, sending...
I20260927 00:48:25.032637 20117 ts_manager.cc:194] Registered new tserver with Master: adfd7d36e15848458663e66414b45bbb (127.0.98.3:41431)
I20260927 00:48:25.033239 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41971
I20260927 00:48:25.036715 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:25.042876 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.039160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.98.62:43369)
0927 00:48:25.039287 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.039294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.039445 (+ 151us) server_negotiation.cc:407] Connection header received
0927 00:48:25.039549 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.039551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.039595 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.039657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:25.039933 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.040465 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.041339 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.041546 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.041637 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.042494 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:25.042497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:25.042499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:25.042509 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:25.042528 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:25.042535 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:25.042600 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.042673 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.042719 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:25.045758 20117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:25.049048 20117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:25.064917 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.058371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.98.3:41431)
0927 00:48:25.059649 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.059663 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.059690 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:48:25.059774 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.059780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.060006 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.060219 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:25.060634 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.061951 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.063586 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064160 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064297 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064512 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.064633 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.064736 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":139,"threads_started":1}
I20260927 00:48:25.065136 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.059051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39681 (local address 127.0.0.1:43352)
0927 00:48:25.059649 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.059664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.059751 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.060651 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.060655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.060672 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:25.060743 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.060749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.062224 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.062227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.063117 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.063127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064586 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.064771 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064796 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.064879 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.064960 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260927 00:48:25.065528 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.058383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37065 (local address 127.0.0.1:36164)
0927 00:48:25.059353 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.059368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.059463 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.060252 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.060255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.060273 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:25.060507 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.060516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.061676 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.061683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.063264 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.063292 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064586 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.064910 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064932 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.064998 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.065254 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":253,"thread_start_us":122,"threads_started":1}
I20260927 00:48:25.065578 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.059502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.0.98.2:39681)
0927 00:48:25.059975 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.059994 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.060021 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:48:25.060106 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.060113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.060351 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.060526 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:25.060866 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.062104 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.063255 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064188 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064317 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064512 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.065219 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.065387 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":113,"threads_started":1}
I20260927 00:48:25.065733 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.058231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41431 (local address 127.0.0.1:38374)
0927 00:48:25.059083 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.059142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.059274 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.060228 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.060235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.060261 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:25.060489 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.060498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.062106 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.062110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.063460 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.063467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064301 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.064418 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064457 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.064584 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.064716 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":86,"thread_start_us":76,"threads_started":1}
I20260927 00:48:25.066351 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.058927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.0.98.1:37065)
0927 00:48:25.059771 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.059782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.059801 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:48:25.059868 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.059873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.060024 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.060443 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:25.060616 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.061526 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.063418 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.064211 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.064317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.064512 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.066087 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.066190 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":117,"threads_started":1}
I20260927 00:48:25.068717 20371 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc7ddef7487424ea623110d686d73f4 (DEFAULT_TABLE table=test-table [id=e911e77da3bc4adbbf40176b7770e3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:25.069443 20502 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc7ddef7487424ea623110d686d73f4 (DEFAULT_TABLE table=test-table [id=e911e77da3bc4adbbf40176b7770e3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:25.069443 20240 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc7ddef7487424ea623110d686d73f4 (DEFAULT_TABLE table=test-table [id=e911e77da3bc4adbbf40176b7770e3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:25.070106 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc7ddef7487424ea623110d686d73f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:25.070608 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc7ddef7487424ea623110d686d73f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:25.070608 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc7ddef7487424ea623110d686d73f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:25.076820 20585 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Bootstrap starting.
I20260927 00:48:25.078259 20586 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Bootstrap starting.
I20260927 00:48:25.079298 20585 tablet_bootstrap.cc:654] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:25.080152 20585 log.cc:824] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:25.080372 20586 tablet_bootstrap.cc:654] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:25.080879 20587 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Bootstrap starting.
I20260927 00:48:25.081178 20586 log.cc:824] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:25.082922 20585 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: No bootstrap required, opened a new log
I20260927 00:48:25.083035 20587 tablet_bootstrap.cc:654] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:25.083105 20586 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: No bootstrap required, opened a new log
I20260927 00:48:25.083164 20585 ts_tablet_manager.cc:1397] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 00:48:25.083289 20586 ts_tablet_manager.cc:1397] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:48:25.083884 20587 log.cc:824] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:25.085848 20587 tablet_bootstrap.cc:492] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: No bootstrap required, opened a new log
I20260927 00:48:25.086028 20587 ts_tablet_manager.cc:1397] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:48:25.086922 20585 raft_consensus.cc:348] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.087186 20585 raft_consensus.cc:374] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:25.087244 20585 raft_consensus.cc:729] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffaabc33c9ed4066beddf05055b82f17, State: Initialized, Role: FOLLOWER
I20260927 00:48:25.087433 20586 raft_consensus.cc:348] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.087693 20586 raft_consensus.cc:374] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:25.087754 20586 raft_consensus.cc:729] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff458041ca04441fa739f677fdd5de7c, State: Initialized, Role: FOLLOWER
I20260927 00:48:25.087779 20585 consensus_queue.cc:260] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.088245 20586 consensus_queue.cc:260] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.088937 20585 ts_tablet_manager.cc:1428] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 00:48:25.089318 20437 heartbeater.cc:503] Master 127.0.98.62:43369 was elected leader, sending a full tablet report...
I20260927 00:48:25.089152 20586 ts_tablet_manager.cc:1428] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:48:25.090312 20306 heartbeater.cc:503] Master 127.0.98.62:43369 was elected leader, sending a full tablet report...
I20260927 00:48:25.090154 20587 raft_consensus.cc:348] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.090399 20587 raft_consensus.cc:374] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:25.090452 20587 raft_consensus.cc:729] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfd7d36e15848458663e66414b45bbb, State: Initialized, Role: FOLLOWER
I20260927 00:48:25.090871 20587 consensus_queue.cc:260] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.091457 20568 heartbeater.cc:503] Master 127.0.98.62:43369 was elected leader, sending a full tablet report...
I20260927 00:48:25.091791 20587 ts_tablet_manager.cc:1428] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 00:48:25.133003 20307 tablet.cc:2369] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:25.270257 20569 tablet.cc:2369] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:48:25.334147 20438 tablet.cc:2369] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:25.350015 20593 raft_consensus.cc:482] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:25.350204 20593 raft_consensus.cc:504] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.351269 20593 leader_election.cc:288] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff458041ca04441fa739f677fdd5de7c (127.0.98.1:37065), ffaabc33c9ed4066beddf05055b82f17 (127.0.98.2:39681)
I20260927 00:48:25.355505 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.351666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56953 (local address 127.0.98.1:37065)
0927 00:48:25.351819 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.351824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.351841 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:48:25.351908 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.351911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.351970 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.352071 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:25.352364 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.353109 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.354387 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.354991 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.355073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.355123 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.355354 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.355407 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:48:25.355520 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.351545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37065 (local address 127.0.98.3:56953)
0927 00:48:25.351707 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.351720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.351822 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.352090 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.352094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.352114 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:25.352246 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.352252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.353230 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.353233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.354259 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.354268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.355106 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.355110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.355220 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.355246 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.355324 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.355393 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:48:25.356220 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.352004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35331 (local address 127.0.98.2:39681)
0927 00:48:25.352147 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.352151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.352392 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:48:25.352495 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.352497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.352540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.352599 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:25.352949 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.353686 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.354954 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.355579 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.355684 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.355754 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.356063 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.356133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:48:25.356266 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.351858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39681 (local address 127.0.98.3:35331)
0927 00:48:25.352275 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.352286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:25.352400 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:25.352640 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:25.352643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:25.352661 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:25.352798 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.352803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.353831 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.353835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.354812 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:25.354824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.355805 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.355809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:25.355910 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.355937 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:48:25.356031 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:48:25.356137 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":116,"threads_started":1}
I20260927 00:48:25.356235 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc7ddef7487424ea623110d686d73f4" candidate_uuid: "adfd7d36e15848458663e66414b45bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff458041ca04441fa739f677fdd5de7c" is_pre_election: true
I20260927 00:48:25.356561 20260 raft_consensus.cc:2454] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adfd7d36e15848458663e66414b45bbb in term 0.
I20260927 00:48:25.356881 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc7ddef7487424ea623110d686d73f4" candidate_uuid: "adfd7d36e15848458663e66414b45bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffaabc33c9ed4066beddf05055b82f17" is_pre_election: true
I20260927 00:48:25.357112 20391 raft_consensus.cc:2454] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adfd7d36e15848458663e66414b45bbb in term 0.
I20260927 00:48:25.357116 20457 leader_election.cc:302] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adfd7d36e15848458663e66414b45bbb, ff458041ca04441fa739f677fdd5de7c; no voters:
I20260927 00:48:25.357417 20593 raft_consensus.cc:2781] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:25.357508 20593 raft_consensus.cc:482] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:25.357553 20593 raft_consensus.cc:3037] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:25.358605 20593 raft_consensus.cc:504] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.358927 20593 leader_election.cc:288] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [CANDIDATE]: Term 1 election: Requested vote from peers ff458041ca04441fa739f677fdd5de7c (127.0.98.1:37065), ffaabc33c9ed4066beddf05055b82f17 (127.0.98.2:39681)
I20260927 00:48:25.359225 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc7ddef7487424ea623110d686d73f4" candidate_uuid: "adfd7d36e15848458663e66414b45bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff458041ca04441fa739f677fdd5de7c"
I20260927 00:48:25.359297 20391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc7ddef7487424ea623110d686d73f4" candidate_uuid: "adfd7d36e15848458663e66414b45bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffaabc33c9ed4066beddf05055b82f17"
I20260927 00:48:25.359350 20260 raft_consensus.cc:3037] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:25.359436 20391 raft_consensus.cc:3037] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:25.360399 20260 raft_consensus.cc:2454] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adfd7d36e15848458663e66414b45bbb in term 1.
I20260927 00:48:25.360401 20391 raft_consensus.cc:2454] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adfd7d36e15848458663e66414b45bbb in term 1.
I20260927 00:48:25.360702 20457 leader_election.cc:302] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adfd7d36e15848458663e66414b45bbb, ff458041ca04441fa739f677fdd5de7c; no voters:
I20260927 00:48:25.360888 20593 raft_consensus.cc:2781] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:25.361116 20593 raft_consensus.cc:686] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 1 LEADER]: Becoming Leader. State: Replica: adfd7d36e15848458663e66414b45bbb, State: Running, Role: LEADER
I20260927 00:48:25.361426 20593 consensus_queue.cc:237] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } }
I20260927 00:48:25.364527 20118 catalog_manager.cc:5495] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb reported cstate change: term changed from 0 to 1, leader changed from <none> to adfd7d36e15848458663e66414b45bbb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "adfd7d36e15848458663e66414b45bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd7d36e15848458663e66414b45bbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 } health_report { overall_health: UNKNOWN } } }
I20260927 00:48:25.427839 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.424582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.98.3:41431)
0927 00:48:25.424725 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:25.424728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:25.424742 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:48:25.424817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:25.424819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:25.424870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:25.424958 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:25.425194 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.425677 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.426528 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:25.426750 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:25.426829 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:25.427072 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:25.427505 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:25.427595 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:25.427686 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 00:48:25.427730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:48:25.433698 20391 raft_consensus.cc:1261] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 1 FOLLOWER]: Refusing update from remote peer adfd7d36e15848458663e66414b45bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:25.433698 20260 raft_consensus.cc:1261] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 1 FOLLOWER]: Refusing update from remote peer adfd7d36e15848458663e66414b45bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:25.434489 20595 consensus_queue.cc:1035] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff458041ca04441fa739f677fdd5de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:25.434721 20593 consensus_queue.cc:1035] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffaabc33c9ed4066beddf05055b82f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:25.442761 20600 mvcc.cc:205] Tried to move back new op lower bound from 7333765551848726528 to 7333765551563239424. Current Snapshot: MvccSnapshot[applied={T|T < 7333765551848726528}]
I20260927 00:48:25.450021 20111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:25.450169 20111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:25.452670 20111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc784f8949b841a999dea7fefd93ab8f: Sending DeleteTablet for 3 replicas of tablet 5bc7ddef7487424ea623110d686d73f4
I20260927 00:48:25.453444 20502 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc7ddef7487424ea623110d686d73f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:25 UTC) from {username='slave'} at 127.0.0.1:38374
I20260927 00:48:25.453758 20240 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc7ddef7487424ea623110d686d73f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:25 UTC) from {username='slave'} at 127.0.0.1:36164
I20260927 00:48:25.454002 20607 tablet_replica.cc:330] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: stopping tablet replica
I20260927 00:48:25.454131 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc7ddef7487424ea623110d686d73f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:25 UTC) from {username='slave'} at 127.0.0.1:43352
I20260927 00:48:25.454285 20608 tablet_replica.cc:330] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c: stopping tablet replica
I20260927 00:48:25.454394 20607 raft_consensus.cc:2229] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:48:25.454591 20608 raft_consensus.cc:2229] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:25.454828 20607 raft_consensus.cc:2258] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:25.454917 20608 raft_consensus.cc:2258] T 5bc7ddef7487424ea623110d686d73f4 P ff458041ca04441fa739f677fdd5de7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:25.454970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20178
I20260927 00:48:25.455724 20609 tablet_replica.cc:330] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: stopping tablet replica
I20260927 00:48:25.456075 20609 raft_consensus.cc:2229] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:25.456418 20609 raft_consensus.cc:2258] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:25.458089 20609 ts_tablet_manager.cc:1905] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:25.460294 20607 ts_tablet_manager.cc:1905] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:25.461599 20609 ts_tablet_manager.cc:1918] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:25.461705 20609 log.cc:1196] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-1/wal/wals/5bc7ddef7487424ea623110d686d73f4
I20260927 00:48:25.462067 20609 ts_tablet_manager.cc:1939] T 5bc7ddef7487424ea623110d686d73f4 P ffaabc33c9ed4066beddf05055b82f17: Deleting consensus metadata
I20260927 00:48:25.463219 20607 ts_tablet_manager.cc:1918] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:48:25.463376 20607 log.cc:1196] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Deleting WAL directory at /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469854491772-392-0/minicluster-data/ts-2/wal/wals/5bc7ddef7487424ea623110d686d73f4
I20260927 00:48:25.463727 20607 ts_tablet_manager.cc:1939] T 5bc7ddef7487424ea623110d686d73f4 P adfd7d36e15848458663e66414b45bbb: Deleting consensus metadata
I20260927 00:48:25.464956 20103 catalog_manager.cc:4841] TS adfd7d36e15848458663e66414b45bbb (127.0.98.3:41431): tablet 5bc7ddef7487424ea623110d686d73f4 (table test-table [id=e911e77da3bc4adbbf40176b7770e3dd]) successfully deleted
I20260927 00:48:25.465838 20104 catalog_manager.cc:4841] TS ffaabc33c9ed4066beddf05055b82f17 (127.0.98.2:39681): tablet 5bc7ddef7487424ea623110d686d73f4 (table test-table [id=e911e77da3bc4adbbf40176b7770e3dd]) successfully deleted
W20260927 00:48:25.466729 20104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37065 (error 108)
I20260927 00:48:25.467845 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:25.467505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37065 (local address 127.0.0.1:49040)
0927 00:48:25.467668 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:25.467746 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:48:25.467876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20309
W20260927 00:48:25.471097 20104 catalog_manager.cc:4584] TS ff458041ca04441fa739f677fdd5de7c (127.0.98.1:37065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bc7ddef7487424ea623110d686d73f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37065: connect: Connection refused (error 111)
I20260927 00:48:25.474906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20440
I20260927 00:48:25.481666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20087
2026-09-27T00:48:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1384 ms)
[----------] 6 tests from EncryptionPolicyTest (11115 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:25 dist-test-slave-l5sb krb5kdc[20613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:25 dist-test-slave-l5sb krb5kdc[20613](info): set up 2 sockets
Sep 27 00:48:25 dist-test-slave-l5sb krb5kdc[20613](info): commencing operation
krb5kdc: starting...
W20260927 00:48:26.510066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:26.629907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:26.735024 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:26.735275 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:26.735313 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:26.739279 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:26.739329 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:26.739348 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:26.739364 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:26.739378 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:26.744617 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:26.745591 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:26.746587 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:26.754770 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:26.754782 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:26.754763 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:26.754848 20626 server_base.cc:1034] running on GCE node
I20260927 00:48:26.755545 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:26.756587 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:26.757740 20626 hybrid_clock.cc:648] HybridClock initialized: now 1790470106757722 us; error 30 us; skew 500 ppm
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:26.760448 20626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:26.761312 20626 webserver.cc:460] Webserver started at http://127.0.0.1:44111/ using document root <none> and password file <none>
I20260927 00:48:26.761914 20626 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:26.762003 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:26.762223 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:26.764170 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "179614cc845347cebe57df5ea892248b"
format_stamp: "Formatted at 2026-09-27 00:48:26 on dist-test-slave-l5sb"
server_key: "37b3f39e6b8da290e4dc01b464e07194"
server_key_iv: "5d358a208d2777a958e93d765ebcfe72"
server_key_version: "encryptionkey@0"
I20260927 00:48:26.764734 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "179614cc845347cebe57df5ea892248b"
format_stamp: "Formatted at 2026-09-27 00:48:26 on dist-test-slave-l5sb"
server_key: "37b3f39e6b8da290e4dc01b464e07194"
server_key_iv: "5d358a208d2777a958e93d765ebcfe72"
server_key_version: "encryptionkey@0"
I20260927 00:48:26.768307 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:26.770644 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:26.772218 20626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:48:26.772344 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "179614cc845347cebe57df5ea892248b"
format_stamp: "Formatted at 2026-09-27 00:48:26 on dist-test-slave-l5sb"
server_key: "37b3f39e6b8da290e4dc01b464e07194"
server_key_iv: "5d358a208d2777a958e93d765ebcfe72"
server_key_version: "encryptionkey@0"
I20260927 00:48:26.772465 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:26.797580 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:26.800357 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:26.800508 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:26.807370 20626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33631
I20260927 00:48:26.807397 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33631 every 8 connection(s)
I20260927 00:48:26.808302 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:26.810988 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:26.815462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20626
I20260927 00:48:26.815608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:26.815908 392 external_mini_cluster.cc:1428] Setting key 1d99d9b441a788bacef62b9e4eca5bbe
I20260927 00:48:26.816833 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: Bootstrap starting.
I20260927 00:48:26.819154 20694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:26.819875 20694 log.cc:824] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:26.821761 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: No bootstrap required, opened a new log
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:48:26.826027 20694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260927 00:48:26.826236 20694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:26.826321 20694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179614cc845347cebe57df5ea892248b, State: Initialized, Role: FOLLOWER
I20260927 00:48:26.826843 20694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260927 00:48:26.827023 20694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:26.827090 20694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:26.827193 20694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:26.828284 20694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260927 00:48:26.828672 20694 leader_election.cc:302] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179614cc845347cebe57df5ea892248b; no voters:
I20260927 00:48:26.828999 20694 leader_election.cc:288] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:26.829095 20699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:26.829356 20699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [term 1 LEADER]: Becoming Leader. State: Replica: 179614cc845347cebe57df5ea892248b, State: Running, Role: LEADER
I20260927 00:48:26.829687 20699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } }
I20260927 00:48:26.830237 20694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:26.830895 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179614cc845347cebe57df5ea892248b. Latest consensus state: current_term: 1 leader_uuid: "179614cc845347cebe57df5ea892248b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260927 00:48:26.831028 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:26.831079 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:26.817372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37355 (local address 127.0.0.1:33631)
0927 00:48:26.817840 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:26.817855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:26.817889 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:48:26.818506 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:26.818521 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:26.818852 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:26.819186 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:26.819975 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:26.820975 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:26.822124 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:26.822802 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:26.822974 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:26.825661 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:26.825666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:26.825680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:26.825705 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:26.827779 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:26.828267 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:26.828273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:26.828278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:26.828431 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:26.828730 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:26.828733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:26.828735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:26.829242 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:26.829439 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:26.829902 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 00:48:26.830129 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260927 00:48:26.831687 20700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179614cc845347cebe57df5ea892248b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179614cc845347cebe57df5ea892248b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33631 } } }
I20260927 00:48:26.831790 20700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:26.834436 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:48:26.835419 20714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:26.835496 20714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:26.835574 20708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:26.836483 20708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:26.842336 20708 catalog_manager.cc:1347] Generated new cluster ID: ae046ffbeac24f75b05a263e07ee89de
I20260927 00:48:26.842396 20708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:26.845463 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:26.836739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42685 (local address 127.0.0.1:33631)
0927 00:48:26.836838 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:26.836843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:26.837089 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:48:26.837194 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:26.837198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:26.837243 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:26.837328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:26.838107 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:26.838933 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:26.840028 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:26.840394 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:26.840494 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:26.842153 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:26.842156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:26.842159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:26.842193 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:26.843714 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:26.844191 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:26.844194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:26.844196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:26.844276 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:26.844597 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:26.844600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:26.844602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:26.844771 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:26.844921 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:26.845206 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:48:26.845297 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:48:26.874245 20708 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:26.875221 20708 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:26.893087 20708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179614cc845347cebe57df5ea892248b: Generated new TSK 0
I20260927 00:48:26.893903 20708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:26.907508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20626
W20260927 00:48:26.913147 20611 connection.cc:535] client connection to 127.0.0.1:33631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33631): Connection reset by peer (error 104)
2026-09-27T00:48:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20725](info): set up 2 sockets
Sep 27 00:48:26 dist-test-slave-l5sb krb5kdc[20725](info): commencing operation
krb5kdc: starting...
W20260927 00:48:27.941987 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:28 dist-test-slave-l5sb krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:28.062433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:28.171470 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:28.171730 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:28.171772 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:28.175815 20738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:28.175873 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:28.175892 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:28.175948 20738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:28.175980 20738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:28.181257 20738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:28.182597 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:28.183624 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:28.191767 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:28.191777 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:28.191773 20738 server_base.cc:1034] running on GCE node
W20260927 00:48:28.191784 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:28.192487 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:28.193547 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:28.194703 20738 hybrid_clock.cc:648] HybridClock initialized: now 1790470108194686 us; error 29 us; skew 500 ppm
Sep 27 00:48:28 dist-test-slave-l5sb krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:28.197506 20738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:28.198407 20738 webserver.cc:460] Webserver started at http://127.0.0.1:36745/ using document root <none> and password file <none>
I20260927 00:48:28.198943 20738 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:28.199004 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:28.199245 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:28.201210 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "217448d90ea64267b94048066db5613d"
format_stamp: "Formatted at 2026-09-27 00:48:28 on dist-test-slave-l5sb"
server_key: "eb215aab68e17dc86e9c615385aeab7b"
server_key_iv: "b4d24e6ac0b4c998dfd403f3e906d7a6"
server_key_version: "encryptionkey@0"
I20260927 00:48:28.201805 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "217448d90ea64267b94048066db5613d"
format_stamp: "Formatted at 2026-09-27 00:48:28 on dist-test-slave-l5sb"
server_key: "eb215aab68e17dc86e9c615385aeab7b"
server_key_iv: "b4d24e6ac0b4c998dfd403f3e906d7a6"
server_key_version: "encryptionkey@0"
I20260927 00:48:28.205368 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:48:28.207758 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:28.208935 20738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:28.209085 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "217448d90ea64267b94048066db5613d"
format_stamp: "Formatted at 2026-09-27 00:48:28 on dist-test-slave-l5sb"
server_key: "eb215aab68e17dc86e9c615385aeab7b"
server_key_iv: "b4d24e6ac0b4c998dfd403f3e906d7a6"
server_key_version: "encryptionkey@0"
I20260927 00:48:28.209194 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:28.237429 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:28.241050 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:28.241262 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:28.248775 20738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37287
I20260927 00:48:28.248799 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37287 every 8 connection(s)
I20260927 00:48:28.249752 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:28.252429 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:28.257870 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d: Bootstrap starting.
I20260927 00:48:28.258358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20738
I20260927 00:48:28.258500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:28.258793 392 external_mini_cluster.cc:1428] Setting key c10b708142cb57e244b64b79af848151
I20260927 00:48:28.260615 20806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:28.261332 20806 log.cc:824] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:28.263298 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d: No bootstrap required, opened a new log
I20260927 00:48:28.267603 20806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260927 00:48:28.267853 20806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:28.267933 20806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217448d90ea64267b94048066db5613d, State: Initialized, Role: FOLLOWER
I20260927 00:48:28.268057 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:28.260111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55073 (local address 127.0.0.1:37287)
0927 00:48:28.260545 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:28.260560 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:28.260595 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:48:28.261262 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:28.261279 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:28.261699 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:28.262104 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:28.262422 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:28.263375 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:28.264595 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:28.265097 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:28.265221 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:28.266224 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:28.266229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:28.266241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:28.266632 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:28.266687 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:28.266717 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:28.266819 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:28.267058 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:48:28.267208 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260927 00:48:28.268533 20806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260927 00:48:28.268700 20806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:28.268764 20806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:28.268846 20806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:28.269814 20806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260927 00:48:28.270188 20806 leader_election.cc:302] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217448d90ea64267b94048066db5613d; no voters:
I20260927 00:48:28.270520 20806 leader_election.cc:288] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:28.270628 20811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:28.270865 20811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [term 1 LEADER]: Becoming Leader. State: Replica: 217448d90ea64267b94048066db5613d, State: Running, Role: LEADER
I20260927 00:48:28.271049 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:28.271170 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260927 00:48:28.271703 20806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:28.272815 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217448d90ea64267b94048066db5613d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } } }
I20260927 00:48:28.272941 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:28.273164 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217448d90ea64267b94048066db5613d. Latest consensus state: current_term: 1 leader_uuid: "217448d90ea64267b94048066db5613d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217448d90ea64267b94048066db5613d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } } }
I20260927 00:48:28.273257 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:28.273564 20824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:28.276718 20824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:28.278656 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:28.273560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45445 (local address 127.0.0.1:37287)
0927 00:48:28.273966 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:28.273970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:28.274931 (+ 961us) server_negotiation.cc:407] Connection header received
0927 00:48:28.275091 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:28.275094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:28.275145 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:28.275235 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:28.275494 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:28.276026 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:28.276897 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:28.277140 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:28.277256 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:28.278172 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:28.278176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:28.278179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:28.278191 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:28.278223 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:28.278234 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:28.278331 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:28.278478 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:48:28.278515 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:48:28.282938 20824 catalog_manager.cc:1347] Generated new cluster ID: 6edf6134af594ef18db2b7e4c111ae25
I20260927 00:48:28.283016 20824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:28.293934 20824 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:28.294915 20824 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:28.306143 20824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217448d90ea64267b94048066db5613d: Generated new TSK 0
I20260927 00:48:28.306931 20824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:28.314241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20738
W20260927 00:48:28.319037 20723 connection.cc:535] client connection to 127.0.0.1:37287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37287): Connection reset by peer (error 104)
2026-09-27T00:48:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:28 dist-test-slave-l5sb krb5kdc[20837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:28 dist-test-slave-l5sb krb5kdc[20837](info): set up 2 sockets
Sep 27 00:48:28 dist-test-slave-l5sb krb5kdc[20837](info): commencing operation
krb5kdc: starting...
W20260927 00:48:29.365998 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:29 dist-test-slave-l5sb krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:29.484972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:29.593014 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:29.593261 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:29.593340 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:29.597275 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:29.597327 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:29.597347 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:29.597363 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:29.597407 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:29.602627 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:29.603578 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:29.604542 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:29.612951 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:29.612915 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:29.613018 20850 server_base.cc:1034] running on GCE node
W20260927 00:48:29.612931 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:29.613718 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:29.614763 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:29.615926 20850 hybrid_clock.cc:648] HybridClock initialized: now 1790470109615892 us; error 37 us; skew 500 ppm
Sep 27 00:48:29 dist-test-slave-l5sb krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:29.625566 20850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:29.626413 20850 webserver.cc:460] Webserver started at http://127.0.0.1:39785/ using document root <none> and password file <none>
I20260927 00:48:29.626901 20850 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:29.626957 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:29.627121 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:29.628993 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "a95de88225234ad8bca26eba1a4f6d74"
format_stamp: "Formatted at 2026-09-27 00:48:29 on dist-test-slave-l5sb"
server_key: "d963c10b9ad51de9b0c1fdf223ddded9"
server_key_iv: "8d52f3fba10df450b1f95a83b92688fd"
server_key_version: "encryptionkey@0"
I20260927 00:48:29.629490 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "a95de88225234ad8bca26eba1a4f6d74"
format_stamp: "Formatted at 2026-09-27 00:48:29 on dist-test-slave-l5sb"
server_key: "d963c10b9ad51de9b0c1fdf223ddded9"
server_key_iv: "8d52f3fba10df450b1f95a83b92688fd"
server_key_version: "encryptionkey@0"
I20260927 00:48:29.633044 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:29.636054 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:29.637684 20850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:48:29.637854 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "a95de88225234ad8bca26eba1a4f6d74"
format_stamp: "Formatted at 2026-09-27 00:48:29 on dist-test-slave-l5sb"
server_key: "d963c10b9ad51de9b0c1fdf223ddded9"
server_key_iv: "8d52f3fba10df450b1f95a83b92688fd"
server_key_version: "encryptionkey@0"
I20260927 00:48:29.637982 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:29.642515 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:29.645753 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:29.646024 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:29.653393 20850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34315
I20260927 00:48:29.653409 20917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34315 every 8 connection(s)
I20260927 00:48:29.654459 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:29.657398 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:29.660851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20850
I20260927 00:48:29.661005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:29.661300 392 external_mini_cluster.cc:1428] Setting key f349eb21b0ff37c39aebd7d809f7f4f3
I20260927 00:48:29.663439 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74: Bootstrap starting.
I20260927 00:48:29.665748 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:29.666558 20918 log.cc:824] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:29.666610 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:29.662797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44191 (local address 127.0.0.1:34315)
0927 00:48:29.663224 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:29.663239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:29.663273 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:48:29.663963 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:29.663981 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:29.664285 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:29.664622 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:29.665408 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:29.665413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:29.665425 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:29.665435 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:29.665487 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:29.665506 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:29.665622 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:29.665725 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 00:48:29.665913 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
I20260927 00:48:29.668558 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74: No bootstrap required, opened a new log
I20260927 00:48:29.669315 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:29.672849 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:29.671358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41813 (local address 127.0.0.1:34315)
0927 00:48:29.671505 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:29.671511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:29.671602 (+ 91us) server_negotiation.cc:407] Connection header received
0927 00:48:29.671715 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:29.671717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:29.671749 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:29.671826 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:29.672458 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:29.672469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:29.672472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:29.672483 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:29.672510 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:29.672522 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:29.672612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:29.672662 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:48:29.672696 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:48:29.672729 20918 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 00:48:29.673070 20918 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:29.673133 20918 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95de88225234ad8bca26eba1a4f6d74, State: Initialized, Role: FOLLOWER
I20260927 00:48:29.673617 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 00:48:29.673796 20918 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:29.673875 20918 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:29.673985 20918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:29.674858 20918 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 00:48:29.675192 20918 leader_election.cc:302] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95de88225234ad8bca26eba1a4f6d74; no voters:
I20260927 00:48:29.675479 20918 leader_election.cc:288] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:29.675557 20928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:29.675752 20928 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [term 1 LEADER]: Becoming Leader. State: Replica: a95de88225234ad8bca26eba1a4f6d74, State: Running, Role: LEADER
I20260927 00:48:29.676116 20928 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260927 00:48:29.676807 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:29.677805 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95de88225234ad8bca26eba1a4f6d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260927 00:48:29.677942 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:29.678229 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95de88225234ad8bca26eba1a4f6d74. Latest consensus state: current_term: 1 leader_uuid: "a95de88225234ad8bca26eba1a4f6d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95de88225234ad8bca26eba1a4f6d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260927 00:48:29.678329 20937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:29.678332 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:29.681293 20937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:29.686990 20937 catalog_manager.cc:1347] Generated new cluster ID: 2e9820ddca57497f9474d2c1ea31a6a4
I20260927 00:48:29.687060 20937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:29.703997 20937 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:29.704309 20937 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:29.709123 20937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95de88225234ad8bca26eba1a4f6d74: Generated new TSK 0
I20260927 00:48:29.709748 20937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:29.714294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20850
2026-09-27T00:48:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:29 dist-test-slave-l5sb krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:29 dist-test-slave-l5sb krb5kdc[20949](info): set up 2 sockets
Sep 27 00:48:29 dist-test-slave-l5sb krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260927 00:48:30.746017 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:30 dist-test-slave-l5sb krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:30.868498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:30.974524 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:30.974824 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:30.974907 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:30.978986 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:30.979068 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:30.979106 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:30.979168 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:30.979208 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:30.984413 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:30.985433 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:30.986423 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:30.994715 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:30.994705 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:30.994706 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:30.994884 20962 server_base.cc:1034] running on GCE node
I20260927 00:48:30.995536 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:30.996608 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:30.997797 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790470110997755 us; error 24 us; skew 500 ppm
Sep 27 00:48:30 dist-test-slave-l5sb krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:31.000407 20962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:31.001260 20962 webserver.cc:460] Webserver started at http://127.0.0.1:38385/ using document root <none> and password file <none>
I20260927 00:48:31.001871 20962 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:31.001961 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:31.002171 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:31.004175 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "a5ac083cc6af4f189445186876edf6d3"
format_stamp: "Formatted at 2026-09-27 00:48:30 on dist-test-slave-l5sb"
server_key: "46be762cdf4ce1732b92946bb6c9e21f"
server_key_iv: "750f376f1dcc59a7f7a97231538a73ad"
server_key_version: "encryptionkey@0"
I20260927 00:48:31.004726 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5ac083cc6af4f189445186876edf6d3"
format_stamp: "Formatted at 2026-09-27 00:48:30 on dist-test-slave-l5sb"
server_key: "46be762cdf4ce1732b92946bb6c9e21f"
server_key_iv: "750f376f1dcc59a7f7a97231538a73ad"
server_key_version: "encryptionkey@0"
I20260927 00:48:31.008431 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:31.010865 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:31.012037 20962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:48:31.012173 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "a5ac083cc6af4f189445186876edf6d3"
format_stamp: "Formatted at 2026-09-27 00:48:30 on dist-test-slave-l5sb"
server_key: "46be762cdf4ce1732b92946bb6c9e21f"
server_key_iv: "750f376f1dcc59a7f7a97231538a73ad"
server_key_version: "encryptionkey@0"
I20260927 00:48:31.012283 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:31.023625 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:31.026283 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:31.026469 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:31.032908 20962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37149
I20260927 00:48:31.032987 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37149 every 8 connection(s)
I20260927 00:48:31.033911 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:31.036566 21030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:31.041749 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: Bootstrap starting.
I20260927 00:48:31.044117 21030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:31.044175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 20962
I20260927 00:48:31.044299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:31.044616 392 external_mini_cluster.cc:1428] Setting key 6c945c06f566cb5901b8be419ce3c835
I20260927 00:48:31.045047 21030 log.cc:824] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:31.047109 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: No bootstrap required, opened a new log
I20260927 00:48:31.050505 21030 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260927 00:48:31.050725 21030 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:31.050797 21030 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ac083cc6af4f189445186876edf6d3, State: Initialized, Role: FOLLOWER
I20260927 00:48:31.051312 21030 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260927 00:48:31.051489 21030 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:31.051566 21030 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:31.051688 21030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:31.052713 21030 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
Sep 27 00:48:31 dist-test-slave-l5sb krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:48:31.053062 21030 leader_election.cc:302] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ac083cc6af4f189445186876edf6d3; no voters:
I20260927 00:48:31.053454 21030 leader_election.cc:288] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:31.053495 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:31.053705 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [term 1 LEADER]: Becoming Leader. State: Replica: a5ac083cc6af4f189445186876edf6d3, State: Running, Role: LEADER
I20260927 00:48:31.054023 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260927 00:48:31.054824 21030 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:31.056008 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ac083cc6af4f189445186876edf6d3. Latest consensus state: current_term: 1 leader_uuid: "a5ac083cc6af4f189445186876edf6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260927 00:48:31.056150 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:31.056478 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ac083cc6af4f189445186876edf6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ac083cc6af4f189445186876edf6d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260927 00:48:31.056596 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:31.059402 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:31.045729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.0.0.1:37149)
0927 00:48:31.046177 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:31.046192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:31.046225 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:48:31.046858 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:31.046874 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:31.047233 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:31.047583 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:31.048327 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:31.049313 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:31.050233 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:31.050727 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:31.050910 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:31.053726 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:31.053734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:31.053754 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:31.053817 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:31.055894 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:31.056524 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:31.056532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:31.056537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:31.056764 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:31.057236 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:31.057239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:31.057242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:31.057721 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:31.057960 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:31.058208 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:48:31.058508 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
W20260927 00:48:31.060118 21050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:31.060189 21050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:31.060243 21044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:31.060941 21044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:31.062446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:31.066665 21044 catalog_manager.cc:1347] Generated new cluster ID: 21788b6c1aa349d8ba46cf2ab66f0b47
I20260927 00:48:31.066721 21044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:31.076296 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:31.064377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52949 (local address 127.0.0.1:37149)
0927 00:48:31.064481 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:31.064485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:31.064719 (+ 234us) server_negotiation.cc:407] Connection header received
0927 00:48:31.064830 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:31.064833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:31.064871 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:31.064959 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:31.065683 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:31.066506 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:31.070930 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:31.071315 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:31.071427 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:31.072927 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:31.072931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:31.072934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:31.072959 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:31.074624 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:31.075069 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:31.075073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:31.075075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:31.075169 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:31.075426 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:31.075429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:31.075431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:31.075640 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:31.075766 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:31.075991 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 00:48:31.076115 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:48:31.082923 21044 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:31.084196 21044 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:31.093535 21044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ac083cc6af4f189445186876edf6d3: Generated new TSK 0
I20260927 00:48:31.094398 21044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:31.101511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 20962
2026-09-27T00:48:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:31 dist-test-slave-l5sb krb5kdc[21061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:31 dist-test-slave-l5sb krb5kdc[21061](info): set up 2 sockets
Sep 27 00:48:31 dist-test-slave-l5sb krb5kdc[21061](info): commencing operation
krb5kdc: starting...
W20260927 00:48:32.130072 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:32 dist-test-slave-l5sb krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:32.256692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:32.368288 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:32.368575 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:32.368649 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:32.372691 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:32.372774 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:32.372812 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:32.372864 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:32.372900 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:32.378135 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:32.379174 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:32.380191 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:32.388585 21074 server_base.cc:1034] running on GCE node
W20260927 00:48:32.388537 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:32.388531 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:32.388537 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:32.389387 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:32.390489 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:32.391652 21074 hybrid_clock.cc:648] HybridClock initialized: now 1790470112391640 us; error 25 us; skew 500 ppm
Sep 27 00:48:32 dist-test-slave-l5sb krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:32.394390 21074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:32.395504 21074 webserver.cc:460] Webserver started at http://127.0.0.1:32971/ using document root <none> and password file <none>
I20260927 00:48:32.396199 21074 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:32.396265 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:32.396422 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:32.398461 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "9500a81beae64224803449e30a99303e"
format_stamp: "Formatted at 2026-09-27 00:48:32 on dist-test-slave-l5sb"
server_key: "ca1e3191d1bb3c0d354683b813a4160b"
server_key_iv: "3ff345db970513efd3fd850e4ab0b06e"
server_key_version: "encryptionkey@0"
I20260927 00:48:32.398980 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "9500a81beae64224803449e30a99303e"
format_stamp: "Formatted at 2026-09-27 00:48:32 on dist-test-slave-l5sb"
server_key: "ca1e3191d1bb3c0d354683b813a4160b"
server_key_iv: "3ff345db970513efd3fd850e4ab0b06e"
server_key_version: "encryptionkey@0"
I20260927 00:48:32.402729 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:48:32.405280 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:32.406663 21074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:48:32.406810 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "9500a81beae64224803449e30a99303e"
format_stamp: "Formatted at 2026-09-27 00:48:32 on dist-test-slave-l5sb"
server_key: "ca1e3191d1bb3c0d354683b813a4160b"
server_key_iv: "3ff345db970513efd3fd850e4ab0b06e"
server_key_version: "encryptionkey@0"
I20260927 00:48:32.406940 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:32.434631 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:32.438196 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:32.438411 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:32.445441 21074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39907
I20260927 00:48:32.445468 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39907 every 8 connection(s)
I20260927 00:48:32.446384 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:32.449023 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:32.453080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21074
I20260927 00:48:32.453259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:32.453536 392 external_mini_cluster.cc:1428] Setting key e0341bbbfb9116271f6ca992398e3c21
I20260927 00:48:32.455533 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: Bootstrap starting.
I20260927 00:48:32.458561 21142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:32.459455 21142 log.cc:824] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:32.461658 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: No bootstrap required, opened a new log
I20260927 00:48:32.464522 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:32.455078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37397 (local address 127.0.0.1:39907)
0927 00:48:32.455507 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:32.455527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:32.455573 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:48:32.456520 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:32.456542 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:32.457021 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:32.457406 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:32.457852 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:32.459502 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:32.460624 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:32.461352 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:32.461527 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:32.462389 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:32.462396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:32.462411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:32.462923 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:32.463005 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:32.463037 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:32.463142 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:32.463297 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 00:48:32.463511 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260927 00:48:32.466358 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260927 00:48:32.466679 21142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:32.466759 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9500a81beae64224803449e30a99303e, State: Initialized, Role: FOLLOWER
I20260927 00:48:32.467278 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260927 00:48:32.467458 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:32.467531 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:32.467612 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:32.467736 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:32.468572 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260927 00:48:32.468946 21142 leader_election.cc:302] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9500a81beae64224803449e30a99303e; no voters:
I20260927 00:48:32.469261 21142 leader_election.cc:288] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:32.469353 21151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:32.469650 21151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [term 1 LEADER]: Becoming Leader. State: Replica: 9500a81beae64224803449e30a99303e, State: Running, Role: LEADER
I20260927 00:48:32.469964 21151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260927 00:48:32.470683 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:32.471580 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9500a81beae64224803449e30a99303e. Latest consensus state: current_term: 1 leader_uuid: "9500a81beae64224803449e30a99303e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260927 00:48:32.471692 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:32.472029 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9500a81beae64224803449e30a99303e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9500a81beae64224803449e30a99303e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260927 00:48:32.472122 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:32.475009 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:32.469732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48599 (local address 127.0.0.1:39907)
0927 00:48:32.469892 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:32.469896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:32.470425 (+ 529us) server_negotiation.cc:407] Connection header received
0927 00:48:32.470526 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:32.470529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:32.470575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:32.470660 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:32.471020 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:32.471918 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:32.473237 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:32.473587 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:32.473647 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:32.474579 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:32.474583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:32.474586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:32.474597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:32.474623 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:32.474642 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:32.474716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:32.474807 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 00:48:32.474854 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":94}
W20260927 00:48:32.476248 21168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:32.476334 21168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:32.476406 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:32.477110 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:32.482489 21161 catalog_manager.cc:1347] Generated new cluster ID: 397600e701a946328d2686172c9c4463
I20260927 00:48:32.482558 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:32.499671 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:32.500991 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:32.506946 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9500a81beae64224803449e30a99303e: Generated new TSK 0
I20260927 00:48:32.507771 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:32.512135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21074
2026-09-27T00:48:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:32 dist-test-slave-l5sb krb5kdc[21173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:32 dist-test-slave-l5sb krb5kdc[21173](info): set up 2 sockets
Sep 27 00:48:32 dist-test-slave-l5sb krb5kdc[21173](info): commencing operation
krb5kdc: starting...
W20260927 00:48:33.562218 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:33 dist-test-slave-l5sb krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:33.682806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:33.791356 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:33.791604 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:33.791646 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:33.795581 21186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:33.795632 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:33.795651 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:33.795667 21186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:33.795712 21186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:33.800972 21186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:33.801962 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:33.802938 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:33.811493 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:33.811517 21186 server_base.cc:1034] running on GCE node
W20260927 00:48:33.811470 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:33.811470 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:33.812256 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:33.813333 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:33.814550 21186 hybrid_clock.cc:648] HybridClock initialized: now 1790470113814533 us; error 37 us; skew 500 ppm
Sep 27 00:48:33 dist-test-slave-l5sb krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:33.817397 21186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:33.818352 21186 webserver.cc:460] Webserver started at http://127.0.0.1:42655/ using document root <none> and password file <none>
I20260927 00:48:33.818931 21186 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:33.819027 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:33.819255 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:33.821251 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "94a677954970483f892ef419bb9772e3"
format_stamp: "Formatted at 2026-09-27 00:48:33 on dist-test-slave-l5sb"
server_key: "afc64d2153a4e99d6f582afc91cb6a52"
server_key_iv: "79c704d9098456eff952887fd4eb71c8"
server_key_version: "encryptionkey@0"
I20260927 00:48:33.821884 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "94a677954970483f892ef419bb9772e3"
format_stamp: "Formatted at 2026-09-27 00:48:33 on dist-test-slave-l5sb"
server_key: "afc64d2153a4e99d6f582afc91cb6a52"
server_key_iv: "79c704d9098456eff952887fd4eb71c8"
server_key_version: "encryptionkey@0"
I20260927 00:48:33.825477 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:48:33.827934 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:33.829119 21186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:33.829272 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "94a677954970483f892ef419bb9772e3"
format_stamp: "Formatted at 2026-09-27 00:48:33 on dist-test-slave-l5sb"
server_key: "afc64d2153a4e99d6f582afc91cb6a52"
server_key_iv: "79c704d9098456eff952887fd4eb71c8"
server_key_version: "encryptionkey@0"
I20260927 00:48:33.829391 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:33.833962 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:33.836777 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:33.836962 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:33.843499 21186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43801
I20260927 00:48:33.843565 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43801 every 8 connection(s)
I20260927 00:48:33.844434 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:33.847110 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:33.848595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21186
I20260927 00:48:33.848762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:33.849021 392 external_mini_cluster.cc:1428] Setting key 85ec670b798ec3b7457200d6bbe14078
I20260927 00:48:33.853066 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3: Bootstrap starting.
I20260927 00:48:33.854077 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:33.850381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49339 (local address 127.0.0.1:43801)
0927 00:48:33.850779 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:33.850793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:33.850827 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:48:33.851491 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:33.851507 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:33.851766 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:33.852093 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:33.852856 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:33.852862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:33.852883 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:33.852893 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:33.852969 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:33.852988 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:33.853139 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:33.853208 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:48:33.853356 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260927 00:48:33.855538 21254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:33.856405 21254 log.cc:824] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:33.856983 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:33.858536 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3: No bootstrap required, opened a new log
I20260927 00:48:33.860737 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:33.859211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39911 (local address 127.0.0.1:43801)
0927 00:48:33.859323 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:33.859327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:33.859519 (+ 192us) server_negotiation.cc:407] Connection header received
0927 00:48:33.859600 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:33.859603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:33.859637 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:33.859726 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:33.860375 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:33.860379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:33.860381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:33.860393 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:33.860420 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:33.860432 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:33.860518 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:33.860586 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:48:33.860621 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:48:33.862986 21254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } }
I20260927 00:48:33.863202 21254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:33.863235 21254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a677954970483f892ef419bb9772e3, State: Initialized, Role: FOLLOWER
I20260927 00:48:33.863646 21254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } }
I20260927 00:48:33.863765 21254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:33.863811 21254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:33.863879 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:33.864722 21254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } }
I20260927 00:48:33.865006 21254 leader_election.cc:302] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94a677954970483f892ef419bb9772e3; no voters:
I20260927 00:48:33.865284 21254 leader_election.cc:288] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:33.865357 21264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:33.865542 21264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [term 1 LEADER]: Becoming Leader. State: Replica: 94a677954970483f892ef419bb9772e3, State: Running, Role: LEADER
I20260927 00:48:33.865840 21264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } }
I20260927 00:48:33.866552 21254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:33.867444 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94a677954970483f892ef419bb9772e3. Latest consensus state: current_term: 1 leader_uuid: "94a677954970483f892ef419bb9772e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } } }
I20260927 00:48:33.867619 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:33.867785 21265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94a677954970483f892ef419bb9772e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a677954970483f892ef419bb9772e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43801 } } }
I20260927 00:48:33.867925 21265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:33.868106 21273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:33.871135 21273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:33.876715 21273 catalog_manager.cc:1347] Generated new cluster ID: 09392c0c108b41739b180694c1500497
I20260927 00:48:33.876791 21273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:33.899690 21273 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:33.900060 21273 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:33.914423 21273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94a677954970483f892ef419bb9772e3: Generated new TSK 0
I20260927 00:48:33.915177 21273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:33.929071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21186
2026-09-27T00:48:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:33 dist-test-slave-l5sb krb5kdc[21285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:33 dist-test-slave-l5sb krb5kdc[21285](info): set up 2 sockets
Sep 27 00:48:33 dist-test-slave-l5sb krb5kdc[21285](info): commencing operation
krb5kdc: starting...
W20260927 00:48:34.994094 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:35.114915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:35.223918 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:35.224176 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:35.224217 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:35.228199 21298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:35.228256 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:35.228276 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:35.228291 21298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:35.228307 21298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:35.233520 21298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:35.234579 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:35.235832 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:35.244297 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:35.244331 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:35.244323 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:35.244655 21298 server_base.cc:1034] running on GCE node
I20260927 00:48:35.245208 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:35.246354 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:35.247512 21298 hybrid_clock.cc:648] HybridClock initialized: now 1790470115247499 us; error 27 us; skew 500 ppm
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:35.258912 21298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:35.260005 21298 webserver.cc:460] Webserver started at http://127.0.0.1:46189/ using document root <none> and password file <none>
I20260927 00:48:35.260622 21298 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:35.260715 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:35.260964 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:35.263336 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "3e5fe959d05d48dc927d7d34961e84f0"
format_stamp: "Formatted at 2026-09-27 00:48:35 on dist-test-slave-l5sb"
server_key: "fafabc8e2b65b5ab7ee1bff6201b716e"
server_key_iv: "21f0a533cc77b8868239e88a9f47a24a"
server_key_version: "encryptionkey@0"
I20260927 00:48:35.263976 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e5fe959d05d48dc927d7d34961e84f0"
format_stamp: "Formatted at 2026-09-27 00:48:35 on dist-test-slave-l5sb"
server_key: "fafabc8e2b65b5ab7ee1bff6201b716e"
server_key_iv: "21f0a533cc77b8868239e88a9f47a24a"
server_key_version: "encryptionkey@0"
I20260927 00:48:35.267899 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:48:35.270438 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:35.271693 21298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:35.271950 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "3e5fe959d05d48dc927d7d34961e84f0"
format_stamp: "Formatted at 2026-09-27 00:48:35 on dist-test-slave-l5sb"
server_key: "fafabc8e2b65b5ab7ee1bff6201b716e"
server_key_iv: "21f0a533cc77b8868239e88a9f47a24a"
server_key_version: "encryptionkey@0"
I20260927 00:48:35.272074 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:35.282263 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:35.285519 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:35.285718 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:35.292760 21298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35941
I20260927 00:48:35.292774 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35941 every 8 connection(s)
I20260927 00:48:35.293704 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:35.296444 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:35.300824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21298
I20260927 00:48:35.301002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:35.301301 392 external_mini_cluster.cc:1428] Setting key d0d096a4014f9f8154cb95dc0a315b44
I20260927 00:48:35.303076 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0: Bootstrap starting.
I20260927 00:48:35.305617 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:35.306418 21366 log.cc:824] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:35.308609 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0: No bootstrap required, opened a new log
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:48:35.313486 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 00:48:35.313822 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:35.313920 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5fe959d05d48dc927d7d34961e84f0, State: Initialized, Role: FOLLOWER
I20260927 00:48:35.314451 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 00:48:35.314631 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:35.314736 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:35.314839 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:35.316126 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 00:48:35.316511 21366 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e5fe959d05d48dc927d7d34961e84f0; no voters:
I20260927 00:48:35.316867 21366 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:35.316984 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:35.317241 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [term 1 LEADER]: Becoming Leader. State: Replica: 3e5fe959d05d48dc927d7d34961e84f0, State: Running, Role: LEADER
I20260927 00:48:35.317314 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:35.302799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:43425 (local address 127.0.0.1:35941)
0927 00:48:35.303245 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:35.303268 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:35.303304 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:35.303947 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:35.303964 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:35.304349 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:35.304643 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:35.305571 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:35.306762 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:35.307926 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:35.308422 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:35.308581 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:35.311449 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:35.311455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:35.311478 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:35.311520 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:35.313634 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:35.314359 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:35.314365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:35.314372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:35.314520 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:35.314969 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:35.314973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:35.314975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:35.315507 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:35.315737 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:35.316031 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 00:48:35.316283 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":170,"threads_started":1}
I20260927 00:48:35.317626 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } }
I20260927 00:48:35.318202 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:35.318923 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5fe959d05d48dc927d7d34961e84f0. Latest consensus state: current_term: 1 leader_uuid: "3e5fe959d05d48dc927d7d34961e84f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260927 00:48:35.319654 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:35.319540 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e5fe959d05d48dc927d7d34961e84f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5fe959d05d48dc927d7d34961e84f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35941 } } }
I20260927 00:48:35.319916 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:35.320241 21381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:35.321138 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:35.323988 21381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:35.330066 21381 catalog_manager.cc:1347] Generated new cluster ID: 49b566d17e814b23bec85f660010d57f
I20260927 00:48:35.330191 21381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:35.333443 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:35.323437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:33251 (local address 127.0.0.1:35941)
0927 00:48:35.323540 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:35.323545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:35.324602 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 00:48:35.324717 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:35.324721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:35.324774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:35.324862 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:35.325694 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:35.326545 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:35.327646 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:35.328086 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:35.328200 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:35.329756 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:35.329759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:35.329762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:35.329814 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:35.331610 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:35.332180 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:35.332183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:35.332185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:35.332262 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:35.332608 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:35.332611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:35.332612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:35.332792 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:35.332909 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:35.333210 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:48:35.333296 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:48:35.343314 21381 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:35.344406 21381 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:35.353691 21381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5fe959d05d48dc927d7d34961e84f0: Generated new TSK 0
I20260927 00:48:35.354425 21381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:35.361702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21298
W20260927 00:48:35.367146 21283 connection.cc:535] client connection to 127.0.0.1:35941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35941): Connection reset by peer (error 104)
2026-09-27T00:48:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21397](info): set up 2 sockets
Sep 27 00:48:35 dist-test-slave-l5sb krb5kdc[21397](info): commencing operation
krb5kdc: starting...
W20260927 00:48:36.410354 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:36 dist-test-slave-l5sb krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:36.528468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:36.634202 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:36.634436 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:36.634478 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:36.638335 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:36.638386 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:36.638404 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:36.638420 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:36.638435 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:36.643474 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:36.644430 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:36.645429 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:36.653558 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:36.653573 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:36.653614 21410 server_base.cc:1034] running on GCE node
W20260927 00:48:36.653563 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:36.654296 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:36.655377 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:36.656559 21410 hybrid_clock.cc:648] HybridClock initialized: now 1790470116656532 us; error 40 us; skew 500 ppm
Sep 27 00:48:36 dist-test-slave-l5sb krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:36.659188 21410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:36.660068 21410 webserver.cc:460] Webserver started at http://127.0.0.1:42901/ using document root <none> and password file <none>
I20260927 00:48:36.660607 21410 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:36.660693 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:36.660921 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:36.662938 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "879d6a2026004cce9f4148a8283b0453"
format_stamp: "Formatted at 2026-09-27 00:48:36 on dist-test-slave-l5sb"
server_key: "c409af741379dac913f99c8faa686c48"
server_key_iv: "6749168803fa5d9489a0a0023e641842"
server_key_version: "encryptionkey@0"
I20260927 00:48:36.663508 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "879d6a2026004cce9f4148a8283b0453"
format_stamp: "Formatted at 2026-09-27 00:48:36 on dist-test-slave-l5sb"
server_key: "c409af741379dac913f99c8faa686c48"
server_key_iv: "6749168803fa5d9489a0a0023e641842"
server_key_version: "encryptionkey@0"
I20260927 00:48:36.667155 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:36.669446 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:36.670708 21410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:36.670828 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "879d6a2026004cce9f4148a8283b0453"
format_stamp: "Formatted at 2026-09-27 00:48:36 on dist-test-slave-l5sb"
server_key: "c409af741379dac913f99c8faa686c48"
server_key_iv: "6749168803fa5d9489a0a0023e641842"
server_key_version: "encryptionkey@0"
I20260927 00:48:36.670946 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:36.695421 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:36.698231 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:36.698380 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:36.705273 21410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46441
I20260927 00:48:36.705297 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46441 every 8 connection(s)
I20260927 00:48:36.706207 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:36.708797 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:36.714154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21410
I20260927 00:48:36.714238 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453: Bootstrap starting.
I20260927 00:48:36.714286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:36.714579 392 external_mini_cluster.cc:1428] Setting key ee23855e3953f0e339d3b6a580424662
I20260927 00:48:36.716710 21478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:36.717427 21478 log.cc:824] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:36.719403 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453: No bootstrap required, opened a new log
I20260927 00:48:36.723443 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:36.715917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:58567 (local address 127.0.0.1:46441)
0927 00:48:36.716313 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:36.716328 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:36.716363 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:48:36.717054 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:36.717071 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:36.717397 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:36.717722 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:36.718165 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:36.719145 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:36.720321 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:36.720767 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:36.720880 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:36.721747 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:36.721752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:36.721763 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:36.722181 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:36.722231 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:36.722253 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:36.722351 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:36.722575 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 00:48:36.722736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260927 00:48:36.723469 21478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260927 00:48:36.723749 21478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:36.723834 21478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879d6a2026004cce9f4148a8283b0453, State: Initialized, Role: FOLLOWER
I20260927 00:48:36.724298 21478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260927 00:48:36.724455 21478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:36.724522 21478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:36.724612 21478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:36.725654 21478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260927 00:48:36.726049 21478 leader_election.cc:302] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879d6a2026004cce9f4148a8283b0453; no voters:
I20260927 00:48:36.726375 21478 leader_election.cc:288] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:36.726403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:36.726469 21483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:36.726657 21483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [term 1 LEADER]: Becoming Leader. State: Replica: 879d6a2026004cce9f4148a8283b0453, State: Running, Role: LEADER
I20260927 00:48:36.726977 21483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260927 00:48:36.727562 21478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:36.728106 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879d6a2026004cce9f4148a8283b0453. Latest consensus state: current_term: 1 leader_uuid: "879d6a2026004cce9f4148a8283b0453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260927 00:48:36.728271 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:36.728523 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "879d6a2026004cce9f4148a8283b0453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879d6a2026004cce9f4148a8283b0453" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260927 00:48:36.728623 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:36.728713 21494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:36.731907 21494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:36.733916 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:36.728515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:36779 (local address 127.0.0.1:46441)
0927 00:48:36.728621 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:36.728625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:36.729224 (+ 599us) server_negotiation.cc:407] Connection header received
0927 00:48:36.729404 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:36.729409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:36.729452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:36.729516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:36.729928 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:36.730655 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:36.731804 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:36.732147 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:36.732241 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:36.733290 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:36.733294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:36.733297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:36.733309 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:36.733333 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:36.733346 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:36.733423 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:36.733729 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 00:48:36.733760 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:48:36.737443 21494 catalog_manager.cc:1347] Generated new cluster ID: a7d31e2e64434fc084097eac650352a4
I20260927 00:48:36.737537 21494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:36.747524 21494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:36.748346 21494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:36.758489 21494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 879d6a2026004cce9f4148a8283b0453: Generated new TSK 0
I20260927 00:48:36.759155 21494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:36.771008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21410
W20260927 00:48:36.777194 21395 connection.cc:535] client connection to 127.0.0.1:46441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46441): Connection reset by peer (error 104)
2026-09-27T00:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:36 dist-test-slave-l5sb krb5kdc[21509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:36 dist-test-slave-l5sb krb5kdc[21509](info): set up 2 sockets
Sep 27 00:48:36 dist-test-slave-l5sb krb5kdc[21509](info): commencing operation
krb5kdc: starting...
W20260927 00:48:37.818051 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:37 dist-test-slave-l5sb krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:37.937230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:38.045678 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:38.046003 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:38.046082 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:38.050060 21522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:38.050153 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:38.050192 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:38.050251 21522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:38.050288 21522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:38.055437 21522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:38.056485 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:38.057497 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:38.065613 21530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:38.065667 21528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:38.065670 21527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:38.065812 21522 server_base.cc:1034] running on GCE node
I20260927 00:48:38.066428 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:38.067482 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:38.068688 21522 hybrid_clock.cc:648] HybridClock initialized: now 1790470118068656 us; error 40 us; skew 500 ppm
Sep 27 00:48:38 dist-test-slave-l5sb krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:38.071520 21522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:38.072487 21522 webserver.cc:460] Webserver started at http://127.0.0.1:36651/ using document root <none> and password file <none>
I20260927 00:48:38.073053 21522 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:38.073114 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:38.073346 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:38.075416 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "efeeb1e8c43b4f199dc54f429e96337d"
format_stamp: "Formatted at 2026-09-27 00:48:38 on dist-test-slave-l5sb"
server_key: "a17f5e72dc968233eda134a175c0b294"
server_key_iv: "9e768ab42eb97fe317d4659e7698932f"
server_key_version: "encryptionkey@0"
I20260927 00:48:38.076004 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "efeeb1e8c43b4f199dc54f429e96337d"
format_stamp: "Formatted at 2026-09-27 00:48:38 on dist-test-slave-l5sb"
server_key: "a17f5e72dc968233eda134a175c0b294"
server_key_iv: "9e768ab42eb97fe317d4659e7698932f"
server_key_version: "encryptionkey@0"
I20260927 00:48:38.079623 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:48:38.081995 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:38.083178 21522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:38.083313 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "efeeb1e8c43b4f199dc54f429e96337d"
format_stamp: "Formatted at 2026-09-27 00:48:38 on dist-test-slave-l5sb"
server_key: "a17f5e72dc968233eda134a175c0b294"
server_key_iv: "9e768ab42eb97fe317d4659e7698932f"
server_key_version: "encryptionkey@0"
I20260927 00:48:38.083413 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:38.087921 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:38.091234 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:38.091423 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:38.098011 21522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43311
I20260927 00:48:38.098019 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43311 every 8 connection(s)
I20260927 00:48:38.098908 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:38.101612 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:38.102694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21522
I20260927 00:48:38.102829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:38.103097 392 external_mini_cluster.cc:1428] Setting key 8b557458f6bca819c78b1e8b5fea98be
I20260927 00:48:38.108215 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d: Bootstrap starting.
I20260927 00:48:38.108570 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:38.104472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:42745 (local address 127.0.0.1:43311)
0927 00:48:38.104897 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:38.104916 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:38.104959 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:48:38.105810 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:38.105832 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:38.106181 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:38.106545 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:38.107200 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:38.107207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:38.107225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:38.107238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:38.107304 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:38.107329 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:38.107464 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:38.107541 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 00:48:38.107707 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260927 00:48:38.110631 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:38.111459 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:38.111485 21590 log.cc:824] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:38.113348 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d: No bootstrap required, opened a new log
I20260927 00:48:38.115399 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:38.113716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:36013 (local address 127.0.0.1:43311)
0927 00:48:38.113890 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:38.113895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:38.114038 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:48:38.114149 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:38.114151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:38.114181 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:38.114252 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:38.114991 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:38.114994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:38.114996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:38.115006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:38.115029 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:38.115039 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:38.115114 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:38.115215 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 00:48:38.115247 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 00:48:38.117807 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } }
I20260927 00:48:38.118062 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:38.118131 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeeb1e8c43b4f199dc54f429e96337d, State: Initialized, Role: FOLLOWER
I20260927 00:48:38.118568 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } }
I20260927 00:48:38.118734 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:38.118800 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:38.118909 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:38.119918 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } }
I20260927 00:48:38.120265 21590 leader_election.cc:302] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efeeb1e8c43b4f199dc54f429e96337d; no voters:
I20260927 00:48:38.120563 21590 leader_election.cc:288] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:38.120725 21600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:38.120987 21600 raft_consensus.cc:686] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [term 1 LEADER]: Becoming Leader. State: Replica: efeeb1e8c43b4f199dc54f429e96337d, State: Running, Role: LEADER
I20260927 00:48:38.121292 21600 consensus_queue.cc:237] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } }
I20260927 00:48:38.121809 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:38.123097 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [sys.catalog]: SysCatalogTable state changed. Reason: New leader efeeb1e8c43b4f199dc54f429e96337d. Latest consensus state: current_term: 1 leader_uuid: "efeeb1e8c43b4f199dc54f429e96337d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } } }
I20260927 00:48:38.123088 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efeeb1e8c43b4f199dc54f429e96337d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efeeb1e8c43b4f199dc54f429e96337d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43311 } } }
I20260927 00:48:38.123235 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:38.123235 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:38.123694 21610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:38.126785 21610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:38.132517 21610 catalog_manager.cc:1347] Generated new cluster ID: c45f207727264814b2ad62fd1b263b67
I20260927 00:48:38.132601 21610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:38.157440 21610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:38.157789 21610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:38.164871 21610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efeeb1e8c43b4f199dc54f429e96337d: Generated new TSK 0
I20260927 00:48:38.165577 21610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:38.174401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21522
2026-09-27T00:48:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:38 dist-test-slave-l5sb krb5kdc[21621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:38 dist-test-slave-l5sb krb5kdc[21621](info): set up 2 sockets
Sep 27 00:48:38 dist-test-slave-l5sb krb5kdc[21621](info): commencing operation
krb5kdc: starting...
W20260927 00:48:39.226418 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:39.349480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:39.459290 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:39.459637 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:39.459728 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:39.463845 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:39.463958 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:39.464008 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:39.464066 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:39.464107 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:39.469569 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:39.470696 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:39.471858 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:39.481133 21634 server_base.cc:1034] running on GCE node
W20260927 00:48:39.481096 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:39.481124 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:39.481096 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:39.482148 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:39.483376 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:39.484592 21634 hybrid_clock.cc:648] HybridClock initialized: now 1790470119484554 us; error 47 us; skew 500 ppm
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:39.487576 21634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:39.488438 21634 webserver.cc:460] Webserver started at http://127.0.0.1:38987/ using document root <none> and password file <none>
I20260927 00:48:39.489032 21634 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:39.489126 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:39.489355 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:39.491467 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "a97f68cbb7024e47ab65b260c9bfbcaa"
format_stamp: "Formatted at 2026-09-27 00:48:39 on dist-test-slave-l5sb"
server_key: "628173fdf923632e58327712c6a03b7b"
server_key_iv: "b04a0ae10693cc59cd89a60fac84595e"
server_key_version: "encryptionkey@0"
I20260927 00:48:39.492022 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "a97f68cbb7024e47ab65b260c9bfbcaa"
format_stamp: "Formatted at 2026-09-27 00:48:39 on dist-test-slave-l5sb"
server_key: "628173fdf923632e58327712c6a03b7b"
server_key_iv: "b04a0ae10693cc59cd89a60fac84595e"
server_key_version: "encryptionkey@0"
I20260927 00:48:39.495829 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:48:39.498158 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:39.499434 21634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:39.499573 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "a97f68cbb7024e47ab65b260c9bfbcaa"
format_stamp: "Formatted at 2026-09-27 00:48:39 on dist-test-slave-l5sb"
server_key: "628173fdf923632e58327712c6a03b7b"
server_key_iv: "b04a0ae10693cc59cd89a60fac84595e"
server_key_version: "encryptionkey@0"
I20260927 00:48:39.499683 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:39.513075 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:39.516129 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:39.516324 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:39.523442 21634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38959
I20260927 00:48:39.523470 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38959 every 8 connection(s)
I20260927 00:48:39.524390 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:39.525609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21634
I20260927 00:48:39.525748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:39.526054 392 external_mini_cluster.cc:1428] Setting key 48ab59d7d309490472185d38ec8a1151
I20260927 00:48:39.527606 21702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:39.534119 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa: Bootstrap starting.
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:48:39.536996 21702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:39.537900 21702 log.cc:824] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:39.540171 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa: No bootstrap required, opened a new log
I20260927 00:48:39.541026 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:39.527504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:34349 (local address 127.0.0.1:38959)
0927 00:48:39.527963 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:39.527982 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:39.528017 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:48:39.528885 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:39.528910 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:39.529405 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:39.529812 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:39.530597 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:39.531614 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:39.532492 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:39.532972 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:39.533138 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:39.536238 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:39.536242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:39.536255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:39.536281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:39.538192 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:39.538720 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:39.538725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:39.538728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:39.538831 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:39.539178 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:39.539180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:39.539181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:39.539556 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:39.539713 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:39.540063 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 00:48:39.540284 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":168,"threads_started":1}
I20260927 00:48:39.544240 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:39.544760 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 00:48:39.544983 21702 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:39.545063 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97f68cbb7024e47ab65b260c9bfbcaa, State: Initialized, Role: FOLLOWER
I20260927 00:48:39.545594 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 00:48:39.545725 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:39.545851 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:39.545976 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:39.547137 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 00:48:39.547510 21702 leader_election.cc:302] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a97f68cbb7024e47ab65b260c9bfbcaa; no voters:
I20260927 00:48:39.547857 21702 leader_election.cc:288] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:39.547955 21712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:39.548166 21712 raft_consensus.cc:686] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [term 1 LEADER]: Becoming Leader. State: Replica: a97f68cbb7024e47ab65b260c9bfbcaa, State: Running, Role: LEADER
I20260927 00:48:39.548497 21712 consensus_queue.cc:237] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } }
I20260927 00:48:39.549131 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:39.550197 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } } }
I20260927 00:48:39.550316 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:39.550710 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97f68cbb7024e47ab65b260c9bfbcaa. Latest consensus state: current_term: 1 leader_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97f68cbb7024e47ab65b260c9bfbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38959 } } }
I20260927 00:48:39.550890 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:39.551263 21721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:39.555258 21721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:39.555923 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:39.546420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:59117 (local address 127.0.0.1:38959)
0927 00:48:39.546541 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:39.546546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:39.546722 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:48:39.546865 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:39.546868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:39.546911 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:39.547017 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:39.547717 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:39.548615 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:39.549686 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:39.550089 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:39.550200 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:39.552110 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:39.552114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:39.552116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:39.552139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:39.553799 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:39.554397 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:39.554401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:39.554404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:39.554499 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:39.554954 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:39.554958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:39.554960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:39.555162 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:39.555312 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:39.555679 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:48:39.555776 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:48:39.560809 21721 catalog_manager.cc:1347] Generated new cluster ID: 1dbaeb68ed134843af36fcee59e3f4fd
I20260927 00:48:39.560869 21721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:39.576639 21721 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:39.577495 21721 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:39.582780 21721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a97f68cbb7024e47ab65b260c9bfbcaa: Generated new TSK 0
I20260927 00:48:39.583433 21721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:39.593205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21634
W20260927 00:48:39.598625 21619 connection.cc:535] client connection to 127.0.0.1:38959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38959): Connection reset by peer (error 104)
2026-09-27T00:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21733](info): set up 2 sockets
Sep 27 00:48:39 dist-test-slave-l5sb krb5kdc[21733](info): commencing operation
krb5kdc: starting...
W20260927 00:48:40.630019 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:40 dist-test-slave-l5sb krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:40.757045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:40.868642 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:40.868901 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:40.868942 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:40.873572 21746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:40.873657 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:40.873695 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:40.873735 21746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:40.873795 21746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:40.881575 21746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:40.882686 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:40.883656 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:40.892338 21746 server_base.cc:1034] running on GCE node
W20260927 00:48:40.892385 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:40.892302 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:40.892297 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:40.893054 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:40.894260 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:40.895458 21746 hybrid_clock.cc:648] HybridClock initialized: now 1790470120895434 us; error 26 us; skew 500 ppm
Sep 27 00:48:40 dist-test-slave-l5sb krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:40.905879 21746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:40.906795 21746 webserver.cc:460] Webserver started at http://127.0.0.1:40107/ using document root <none> and password file <none>
I20260927 00:48:40.907323 21746 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:40.907379 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:40.907549 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:40.909518 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "dc60d8108df14547aef4cb29efea2271"
format_stamp: "Formatted at 2026-09-27 00:48:40 on dist-test-slave-l5sb"
server_key: "82eb94edcc5d3291394b6744e9ad6f13"
server_key_iv: "8e3a76061b09c653f5c35b7d7e0856ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:40.910068 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc60d8108df14547aef4cb29efea2271"
format_stamp: "Formatted at 2026-09-27 00:48:40 on dist-test-slave-l5sb"
server_key: "82eb94edcc5d3291394b6744e9ad6f13"
server_key_iv: "8e3a76061b09c653f5c35b7d7e0856ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:40.913601 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:48:40.916019 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:40.917369 21746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:40.917474 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "dc60d8108df14547aef4cb29efea2271"
format_stamp: "Formatted at 2026-09-27 00:48:40 on dist-test-slave-l5sb"
server_key: "82eb94edcc5d3291394b6744e9ad6f13"
server_key_iv: "8e3a76061b09c653f5c35b7d7e0856ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:40.917562 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:40.928340 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:40.931383 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:40.931548 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:40.938632 21746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41851
I20260927 00:48:40.938655 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41851 every 8 connection(s)
I20260927 00:48:40.939591 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:40.942497 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:40.943264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21746
I20260927 00:48:40.943434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:40.943748 392 external_mini_cluster.cc:1428] Setting key a8c1bec7e67718bb13614d6ec3874539
I20260927 00:48:40.948468 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271: Bootstrap starting.
I20260927 00:48:40.950807 21814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:40.951721 21814 log.cc:824] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:40.953052 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:40.945217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:54009 (local address 127.0.0.1:41851)
0927 00:48:40.945681 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:40.945695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:40.945731 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:40.946456 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:40.946473 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:40.946807 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:40.947165 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:40.947555 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:40.948495 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:40.949383 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:40.949924 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:40.950099 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:40.951144 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:40.951150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:40.951166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:40.951675 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:40.951742 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:40.951774 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:40.951917 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:40.952126 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 00:48:40.952259 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":140,"threads_started":1}
I20260927 00:48:40.953817 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271: No bootstrap required, opened a new log
I20260927 00:48:40.955969 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:40.957680 21814 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } }
I20260927 00:48:40.957983 21814 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:40.958051 21814 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60d8108df14547aef4cb29efea2271, State: Initialized, Role: FOLLOWER
I20260927 00:48:40.958525 21814 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } }
I20260927 00:48:40.958679 21814 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:40.958746 21814 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:40.958854 21814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:40.959995 21814 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } }
I20260927 00:48:40.960341 21814 leader_election.cc:302] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc60d8108df14547aef4cb29efea2271; no voters:
I20260927 00:48:40.960687 21814 leader_election.cc:288] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:40.961032 21825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:40.961256 21825 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [term 1 LEADER]: Becoming Leader. State: Replica: dc60d8108df14547aef4cb29efea2271, State: Running, Role: LEADER
I20260927 00:48:40.961575 21825 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } }
I20260927 00:48:40.962126 21814 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:40.962952 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:40.958103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:43163 (local address 127.0.0.1:41851)
0927 00:48:40.958223 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:40.958228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:40.958448 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:48:40.958590 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:40.958593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:40.958640 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:40.958715 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:40.959015 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:40.959777 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:40.960685 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:40.961069 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:40.961170 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:40.962278 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:40.962282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:40.962285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:40.962297 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:40.962325 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:40.962337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:40.962414 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:40.962745 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 00:48:40.962785 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:48:40.963183 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc60d8108df14547aef4cb29efea2271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } } }
I20260927 00:48:40.963303 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:40.963543 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc60d8108df14547aef4cb29efea2271. Latest consensus state: current_term: 1 leader_uuid: "dc60d8108df14547aef4cb29efea2271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc60d8108df14547aef4cb29efea2271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41851 } } }
I20260927 00:48:40.963627 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:40.963725 21833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:40.966917 21833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:40.972733 21833 catalog_manager.cc:1347] Generated new cluster ID: a3a721569318469c837abb76472cc4ba
I20260927 00:48:40.972808 21833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:40.978183 21833 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:40.979091 21833 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:40.985489 21833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc60d8108df14547aef4cb29efea2271: Generated new TSK 0
I20260927 00:48:40.986187 21833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:40.993652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21746
W20260927 00:48:40.998317 21731 connection.cc:535] client connection to 127.0.0.1:41851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41851): Connection reset by peer (error 104)
2026-09-27T00:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:41 dist-test-slave-l5sb krb5kdc[21845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:41 dist-test-slave-l5sb krb5kdc[21845](info): set up 2 sockets
Sep 27 00:48:41 dist-test-slave-l5sb krb5kdc[21845](info): commencing operation
krb5kdc: starting...
W20260927 00:48:42.022401 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:42 dist-test-slave-l5sb krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:48:42.143149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:42.255798 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:42.256125 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:42.256208 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:42.260295 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:42.260391 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:42.260430 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:42.260469 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:42.260499 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:42.265954 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:42.267076 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:42.268133 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:42.276722 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:42.276799 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:42.276849 21858 server_base.cc:1034] running on GCE node
W20260927 00:48:42.276760 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:42.277534 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:42.278699 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:42.279863 21858 hybrid_clock.cc:648] HybridClock initialized: now 1790470122279841 us; error 33 us; skew 500 ppm
Sep 27 00:48:42 dist-test-slave-l5sb krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:42.282670 21858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:48:42.283527 21858 webserver.cc:460] Webserver started at http://127.0.0.1:34235/ using document root <none> and password file <none>
I20260927 00:48:42.284057 21858 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:42.284119 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:42.284279 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:42.286315 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "2fd395d2c6c64e5abf7ee1d4c1473900"
format_stamp: "Formatted at 2026-09-27 00:48:42 on dist-test-slave-l5sb"
server_key: "45e4760c047b2b539398238882bbf5d8"
server_key_iv: "6df499e7724a714b5f4c0f90eea33f47"
server_key_version: "encryptionkey@0"
I20260927 00:48:42.286818 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fd395d2c6c64e5abf7ee1d4c1473900"
format_stamp: "Formatted at 2026-09-27 00:48:42 on dist-test-slave-l5sb"
server_key: "45e4760c047b2b539398238882bbf5d8"
server_key_iv: "6df499e7724a714b5f4c0f90eea33f47"
server_key_version: "encryptionkey@0"
I20260927 00:48:42.290338 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:42.292667 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:42.293962 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:42.294097 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "2fd395d2c6c64e5abf7ee1d4c1473900"
format_stamp: "Formatted at 2026-09-27 00:48:42 on dist-test-slave-l5sb"
server_key: "45e4760c047b2b539398238882bbf5d8"
server_key_iv: "6df499e7724a714b5f4c0f90eea33f47"
server_key_version: "encryptionkey@0"
I20260927 00:48:42.294209 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:42.299002 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:42.301896 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:42.302086 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:42.309386 21858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38917
I20260927 00:48:42.309398 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38917 every 8 connection(s)
I20260927 00:48:42.310343 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:42.313050 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:42.319103 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900: Bootstrap starting.
I20260927 00:48:42.319360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21858
I20260927 00:48:42.319514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:42.319780 392 external_mini_cluster.cc:1428] Setting key 6fce5c262e510179b9b209a2a891dff2
I20260927 00:48:42.321570 21926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:42.322357 21926 log.cc:824] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:42.324255 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900: No bootstrap required, opened a new log
I20260927 00:48:42.325251 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:42.321211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:50171 (local address 127.0.0.1:38917)
0927 00:48:42.321634 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:42.321655 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:42.321716 (+ 61us) server_negotiation.cc:407] Connection header received
0927 00:48:42.322429 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:42.322446 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:42.322782 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:42.323118 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:42.323938 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:42.323943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:42.323955 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:42.323970 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:42.324021 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:42.324040 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:42.324154 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:42.324307 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:48:42.324574 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260927 00:48:42.328191 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:42.328461 21926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } }
I20260927 00:48:42.328696 21926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:42.328771 21926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd395d2c6c64e5abf7ee1d4c1473900, State: Initialized, Role: FOLLOWER
I20260927 00:48:42.329159 21926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } }
I20260927 00:48:42.329301 21926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:42.329383 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:42.329479 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:42.330499 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } }
I20260927 00:48:42.330847 21926 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd395d2c6c64e5abf7ee1d4c1473900; no voters:
I20260927 00:48:42.331212 21926 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:42.331246 21936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:42.331436 21936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [term 1 LEADER]: Becoming Leader. State: Replica: 2fd395d2c6c64e5abf7ee1d4c1473900, State: Running, Role: LEADER
I20260927 00:48:42.331715 21936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } }
I20260927 00:48:42.332432 21926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:42.333561 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } } }
I20260927 00:48:42.333684 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:42.333941 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:42.330412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:48193 (local address 127.0.0.1:38917)
0927 00:48:42.330547 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:42.330551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:42.330687 (+ 136us) server_negotiation.cc:407] Connection header received
0927 00:48:42.330797 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:42.330800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:42.330835 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:42.330914 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:42.333284 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:42.333287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:42.333290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:42.333305 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:42.333339 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:42.333355 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:42.333459 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:42.333757 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:48:42.333826 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:48:42.334018 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd395d2c6c64e5abf7ee1d4c1473900. Latest consensus state: current_term: 1 leader_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd395d2c6c64e5abf7ee1d4c1473900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38917 } } }
I20260927 00:48:42.334097 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:42.335500 21945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:42.338063 21945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:42.343005 21945 catalog_manager.cc:1347] Generated new cluster ID: 3f1f4c93c9a2478c9a6b1dbd7faab675
I20260927 00:48:42.343076 21945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:42.363554 21945 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:42.363909 21945 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:42.368784 21945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fd395d2c6c64e5abf7ee1d4c1473900: Generated new TSK 0
I20260927 00:48:42.369547 21945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:42.377426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21858
2026-09-27T00:48:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:42 dist-test-slave-l5sb krb5kdc[21957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:42 dist-test-slave-l5sb krb5kdc[21957](info): set up 2 sockets
Sep 27 00:48:42 dist-test-slave-l5sb krb5kdc[21957](info): commencing operation
krb5kdc: starting...
W20260927 00:48:43.422194 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:43.541906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:43.651247 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:43.651522 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:43.651566 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:43.655541 21970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:43.655597 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:43.655617 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:43.655633 21970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:43.655648 21970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:43.661463 21970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:43.662532 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:43.663564 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:43.671851 21975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:43.671844 21978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:43.671932 21970 server_base.cc:1034] running on GCE node
W20260927 00:48:43.671844 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:43.672649 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:43.673813 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:43.675004 21970 hybrid_clock.cc:648] HybridClock initialized: now 1790470123674990 us; error 25 us; skew 500 ppm
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:43.677760 21970 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:43.678634 21970 webserver.cc:460] Webserver started at http://127.0.98.62:42313/ using document root <none> and password file <none>
I20260927 00:48:43.679127 21970 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:43.679183 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:43.679337 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:43.681245 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "d1a1b72b05c34c5e9ed5418497f101db"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-l5sb"
server_key: "07046753ae163f024db88fad0b88324f"
server_key_iv: "e29a03e2f54eb0a78932941c539707ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:43.681731 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1a1b72b05c34c5e9ed5418497f101db"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-l5sb"
server_key: "07046753ae163f024db88fad0b88324f"
server_key_iv: "e29a03e2f54eb0a78932941c539707ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:43.685330 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:48:43.687691 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:43.688905 21970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:48:43.689007 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "d1a1b72b05c34c5e9ed5418497f101db"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-l5sb"
server_key: "07046753ae163f024db88fad0b88324f"
server_key_iv: "e29a03e2f54eb0a78932941c539707ae"
server_key_version: "encryptionkey@0"
I20260927 00:48:43.689093 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:43.705157 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:43.708330 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:43.708487 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:43.715201 21970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35095
I20260927 00:48:43.715229 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35095 every 8 connection(s)
I20260927 00:48:43.716143 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:43.717933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 21970
I20260927 00:48:43.718067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:43.718343 392 external_mini_cluster.cc:1428] Setting key 2d2e4d79843c15286792a58721a21865
I20260927 00:48:43.718973 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:43.724542 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db: Bootstrap starting.
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:43.727295 22038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:43.728008 22038 log.cc:824] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:43.730326 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db: No bootstrap required, opened a new log
I20260927 00:48:43.732977 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:43.719678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44669 (local address 127.0.98.62:35095)
0927 00:48:43.720109 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:43.720122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:43.720155 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:48:43.720776 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:43.720791 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:43.721110 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:43.721427 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:43.722254 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.723252 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.724141 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.724641 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.724802 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:43.727638 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:43.727642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:43.727655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:43.727680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:43.729755 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.730278 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.730283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.730289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.730427 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.730726 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.730729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.730730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.731203 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:43.731404 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:43.731715 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 00:48:43.731943 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20260927 00:48:43.734814 22038 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260927 00:48:43.735085 22038 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:43.735142 22038 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a1b72b05c34c5e9ed5418497f101db, State: Initialized, Role: FOLLOWER
I20260927 00:48:43.735628 22038 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260927 00:48:43.735797 22038 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:43.735867 22038 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:43.735962 22038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:43.736069 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:43.737133 22038 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260927 00:48:43.737517 22038 leader_election.cc:302] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a1b72b05c34c5e9ed5418497f101db; no voters:
I20260927 00:48:43.737907 22038 leader_election.cc:288] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:43.738030 22047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:43.738269 22047 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [term 1 LEADER]: Becoming Leader. State: Replica: d1a1b72b05c34c5e9ed5418497f101db, State: Running, Role: LEADER
I20260927 00:48:43.738607 22047 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260927 00:48:43.739389 22038 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:43.740324 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a1b72b05c34c5e9ed5418497f101db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } } }
I20260927 00:48:43.740445 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:43.740664 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a1b72b05c34c5e9ed5418497f101db. Latest consensus state: current_term: 1 leader_uuid: "d1a1b72b05c34c5e9ed5418497f101db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1b72b05c34c5e9ed5418497f101db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } } }
I20260927 00:48:43.740728 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:43.741077 22057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:43.744521 22057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:43.747723 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:43.738420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40005 (local address 127.0.98.62:35095)
0927 00:48:43.738518 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:43.738522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:43.739059 (+ 537us) server_negotiation.cc:407] Connection header received
0927 00:48:43.739147 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:43.739150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:43.739197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:43.739289 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:43.740144 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.740966 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.742167 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.742389 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.742506 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:43.744107 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:43.744111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:43.744114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:43.744138 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:43.745884 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.746458 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.746463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.746469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.746541 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.746870 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.746873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.746875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.747052 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:43.747202 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:43.747467 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 00:48:43.747561 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:48:43.750332 22057 catalog_manager.cc:1347] Generated new cluster ID: 75725692085f434d9297c6c357d7f7e7
I20260927 00:48:43.750391 22057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:43.768200 22057 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:43.769452 22057 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:43.777360 22057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a1b72b05c34c5e9ed5418497f101db: Generated new TSK 0
I20260927 00:48:43.778225 22057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:43.793620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 21970
W20260927 00:48:43.799125 21955 connection.cc:535] client connection to 127.0.98.62:35095 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35095): Connection reset by peer (error 104)
2026-09-27T00:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[22069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[22069](info): set up 2 sockets
Sep 27 00:48:43 dist-test-slave-l5sb krb5kdc[22069](info): commencing operation
krb5kdc: starting...
W20260927 00:48:44.890234 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:44 dist-test-slave-l5sb krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:45.013188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:45.123713 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:45.123960 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:45.123999 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:45.127843 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:45.127902 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:45.127923 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:45.127938 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:45.127954 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:45.132957 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:45.133911 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:45.134852 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:45.143554 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:45.143522 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:45.143534 22082 server_base.cc:1034] running on GCE node
W20260927 00:48:45.143512 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:45.144371 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:45.145498 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:45.146679 22082 hybrid_clock.cc:648] HybridClock initialized: now 1790470125146656 us; error 35 us; skew 500 ppm
Sep 27 00:48:45 dist-test-slave-l5sb krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:45.149462 22082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:45.150460 22082 webserver.cc:460] Webserver started at http://127.0.98.62:44463/ using document root <none> and password file <none>
I20260927 00:48:45.151016 22082 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:45.151106 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:45.151335 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:45.153472 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "d6af03f151e44f6aa4474f18e53d1e1d"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-l5sb"
server_key: "90a513197c27409e6045212ab1062ec3"
server_key_iv: "5cd50809b65aa44ef16f40e75022764e"
server_key_version: "encryptionkey@0"
I20260927 00:48:45.154187 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6af03f151e44f6aa4474f18e53d1e1d"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-l5sb"
server_key: "90a513197c27409e6045212ab1062ec3"
server_key_iv: "5cd50809b65aa44ef16f40e75022764e"
server_key_version: "encryptionkey@0"
I20260927 00:48:45.157917 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:45.160354 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.161556 22082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:45.161695 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "d6af03f151e44f6aa4474f18e53d1e1d"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-l5sb"
server_key: "90a513197c27409e6045212ab1062ec3"
server_key_iv: "5cd50809b65aa44ef16f40e75022764e"
server_key_version: "encryptionkey@0"
I20260927 00:48:45.161832 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:45.170631 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:45.173444 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:45.173655 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:45.180815 22082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44067
I20260927 00:48:45.180840 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44067 every 8 connection(s)
I20260927 00:48:45.181730 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:45.184443 22150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:45.189581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22082
I20260927 00:48:45.189716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:45.189988 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d: Bootstrap starting.
I20260927 00:48:45.190009 392 external_mini_cluster.cc:1428] Setting key ba8f3933560d6ab44a6f0b009b2c04e9
I20260927 00:48:45.192576 22150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:45.193403 22150 log.cc:824] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:45.195533 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d: No bootstrap required, opened a new log
I20260927 00:48:45.199852 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:48:45.200088 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.191361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46167 (local address 127.0.98.62:44067)
0927 00:48:45.191758 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.191776 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.191816 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:48:45.192696 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.192720 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.193143 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.193515 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:45.193988 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.195369 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.196346 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.197010 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.197168 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.198066 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:45.198074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:45.198096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:45.198671 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:45.198737 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:45.198767 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:45.198897 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.199047 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.199177 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260927 00:48:45.200107 22150 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:45.200152 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6af03f151e44f6aa4474f18e53d1e1d, State: Initialized, Role: FOLLOWER
I20260927 00:48:45.200547 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:48:45.200706 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:45.200793 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:45.200914 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:45.202044 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:48:45.202407 22150 leader_election.cc:302] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6af03f151e44f6aa4474f18e53d1e1d; no voters:
I20260927 00:48:45.202739 22150 leader_election.cc:288] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:45.202833 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:45.203073 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [term 1 LEADER]: Becoming Leader. State: Replica: d6af03f151e44f6aa4474f18e53d1e1d, State: Running, Role: LEADER
I20260927 00:48:45.203162 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:45.203423 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:48:45.203990 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:45.204810 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6af03f151e44f6aa4474f18e53d1e1d. Latest consensus state: current_term: 1 leader_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } } }
I20260927 00:48:45.204921 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:45.205250 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6af03f151e44f6aa4474f18e53d1e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } } }
I20260927 00:48:45.205341 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:45.205608 22168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:45.208835 22168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:45.211876 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.207241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58873 (local address 127.0.98.62:44067)
0927 00:48:45.207364 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.207368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.207934 (+ 566us) server_negotiation.cc:407] Connection header received
0927 00:48:45.208055 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.208061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.208112 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.208187 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:45.208450 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.209105 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.210124 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.210371 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.210485 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.211390 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:45.211394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:45.211397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:45.211409 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:45.211436 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:45.211449 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:45.211535 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.211712 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.211745 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:48:45.214722 22168 catalog_manager.cc:1347] Generated new cluster ID: ccfa937e37bb47d8b7c6f4f7dc62351c
I20260927 00:48:45.214787 22168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:45.232564 22168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:45.233546 22168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:45.243836 22168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6af03f151e44f6aa4474f18e53d1e1d: Generated new TSK 0
I20260927 00:48:45.244788 22168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:45.257812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22082
W20260927 00:48:45.262596 22067 connection.cc:535] client connection to 127.0.98.62:44067 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44067): Connection reset by peer (error 104)
2026-09-27T00:48:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:45 dist-test-slave-l5sb krb5kdc[22181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:45 dist-test-slave-l5sb krb5kdc[22181](info): set up 2 sockets
Sep 27 00:48:45 dist-test-slave-l5sb krb5kdc[22181](info): commencing operation
krb5kdc: starting...
W20260927 00:48:46.318495 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:46 dist-test-slave-l5sb krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:46.436499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:46.544638 22194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:46.544945 22194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:46.545017 22194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:46.549083 22194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:46.549172 22194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:46.549208 22194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:46.549273 22194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:46.549310 22194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:46.554502 22194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:46.555589 22194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:46.556640 22194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:46.565231 22200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:46.565231 22202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:46.565281 22199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:46.565466 22194 server_base.cc:1034] running on GCE node
I20260927 00:48:46.566156 22194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:46.567142 22194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:46.568310 22194 hybrid_clock.cc:648] HybridClock initialized: now 1790470126568294 us; error 25 us; skew 500 ppm
Sep 27 00:48:46 dist-test-slave-l5sb krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:46.579340 22194 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:46.580250 22194 webserver.cc:460] Webserver started at http://127.0.98.62:45823/ using document root <none> and password file <none>
I20260927 00:48:46.580801 22194 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:46.580860 22194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:46.581090 22194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:46.583175 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb"
format_stamp: "Formatted at 2026-09-27 00:48:46 on dist-test-slave-l5sb"
server_key: "8400175175f9603562deb05f51bfa797"
server_key_iv: "47c14c6468fd278dcaa2d1b9d7f814c5"
server_key_version: "encryptionkey@0"
I20260927 00:48:46.583726 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb"
format_stamp: "Formatted at 2026-09-27 00:48:46 on dist-test-slave-l5sb"
server_key: "8400175175f9603562deb05f51bfa797"
server_key_iv: "47c14c6468fd278dcaa2d1b9d7f814c5"
server_key_version: "encryptionkey@0"
I20260927 00:48:46.587468 22194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:48:46.589977 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:46.591177 22194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:46.591332 22194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb"
format_stamp: "Formatted at 2026-09-27 00:48:46 on dist-test-slave-l5sb"
server_key: "8400175175f9603562deb05f51bfa797"
server_key_iv: "47c14c6468fd278dcaa2d1b9d7f814c5"
server_key_version: "encryptionkey@0"
I20260927 00:48:46.591447 22194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:46.595968 22194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:46.599063 22194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:46.599249 22194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:46.606045 22194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34721
I20260927 00:48:46.606067 22261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34721 every 8 connection(s)
I20260927 00:48:46.606922 22194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:46.609475 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:46.612275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22194
I20260927 00:48:46.612401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:46.612658 392 external_mini_cluster.cc:1428] Setting key ae2a3d7b5fd34a1f48f49a757b958dbd
I20260927 00:48:46.615360 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb: Bootstrap starting.
I20260927 00:48:46.617990 22262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:46.618153 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:46.613991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35951 (local address 127.0.98.62:34721)
0927 00:48:46.614405 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:46.614424 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:46.614465 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:48:46.615261 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:46.615282 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:46.615624 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:46.615995 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:46.616734 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:46.616740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:46.616754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:46.616765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:46.616824 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:46.616850 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:46.616987 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:46.617108 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 00:48:46.617245 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260927 00:48:46.618777 22262 log.cc:824] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:46.620879 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb: No bootstrap required, opened a new log
I20260927 00:48:46.620926 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:46.624665 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:46.622995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57601 (local address 127.0.98.62:34721)
0927 00:48:46.623116 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:46.623121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:46.623312 (+ 191us) server_negotiation.cc:407] Connection header received
0927 00:48:46.623422 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:46.623424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:46.623451 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:46.623512 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:46.624257 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:46.624260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:46.624262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:46.624271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:46.624294 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:46.624303 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:46.624382 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:46.624490 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 00:48:46.624521 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:48:46.625049 22262 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260927 00:48:46.625313 22262 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:46.625387 22262 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1c6041dff04e9a91f7d40e3e4e68bb, State: Initialized, Role: FOLLOWER
I20260927 00:48:46.625850 22262 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260927 00:48:46.626008 22262 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:46.626072 22262 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:46.626165 22262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:46.627012 22262 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260927 00:48:46.627334 22262 leader_election.cc:302] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb1c6041dff04e9a91f7d40e3e4e68bb; no voters:
I20260927 00:48:46.627622 22262 leader_election.cc:288] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:46.627816 22272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:46.628093 22272 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [term 1 LEADER]: Becoming Leader. State: Replica: eb1c6041dff04e9a91f7d40e3e4e68bb, State: Running, Role: LEADER
I20260927 00:48:46.628384 22272 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260927 00:48:46.628809 22262 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:46.629957 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb1c6041dff04e9a91f7d40e3e4e68bb. Latest consensus state: current_term: 1 leader_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } } }
I20260927 00:48:46.630084 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:46.630491 22281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:46.630507 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1c6041dff04e9a91f7d40e3e4e68bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } } }
I20260927 00:48:46.630681 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:46.633731 22281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:46.638998 22281 catalog_manager.cc:1347] Generated new cluster ID: b6597d2b7deb4fb085632516d24c68b6
I20260927 00:48:46.639057 22281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:46.649816 22281 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:46.650123 22281 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:46.657342 22281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb1c6041dff04e9a91f7d40e3e4e68bb: Generated new TSK 0
I20260927 00:48:46.658077 22281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:46.670727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22194
2026-09-27T00:48:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:46 dist-test-slave-l5sb krb5kdc[22293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:46 dist-test-slave-l5sb krb5kdc[22293](info): set up 2 sockets
Sep 27 00:48:46 dist-test-slave-l5sb krb5kdc[22293](info): commencing operation
krb5kdc: starting...
W20260927 00:48:47.710059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:47 dist-test-slave-l5sb krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:47.830852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:47.938033 22306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:47.938273 22306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:47.938313 22306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:47.942246 22306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:47.942301 22306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:47.942319 22306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:47.942335 22306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:47.942396 22306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:47.947443 22306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:47.948453 22306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:47.949455 22306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:47.957760 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:47.957813 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:47.957919 22306 server_base.cc:1034] running on GCE node
W20260927 00:48:47.957763 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:47.958640 22306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:47.959729 22306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:47.960963 22306 hybrid_clock.cc:648] HybridClock initialized: now 1790470127960942 us; error 29 us; skew 500 ppm
Sep 27 00:48:47 dist-test-slave-l5sb krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:47.963857 22306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:47.964716 22306 webserver.cc:460] Webserver started at http://127.0.98.62:46179/ using document root <none> and password file <none>
I20260927 00:48:47.965241 22306 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:47.965301 22306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:47.965461 22306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:47.967581 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "7369e1e5a20e493b9a148b6e8512dddc"
format_stamp: "Formatted at 2026-09-27 00:48:47 on dist-test-slave-l5sb"
server_key: "8e81f00647129ec24bdf5b357fcc423e"
server_key_iv: "0c830b81995fb13427d52ad0e94ac9ea"
server_key_version: "encryptionkey@0"
I20260927 00:48:47.968088 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "7369e1e5a20e493b9a148b6e8512dddc"
format_stamp: "Formatted at 2026-09-27 00:48:47 on dist-test-slave-l5sb"
server_key: "8e81f00647129ec24bdf5b357fcc423e"
server_key_iv: "0c830b81995fb13427d52ad0e94ac9ea"
server_key_version: "encryptionkey@0"
I20260927 00:48:47.971982 22306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:48:47.974443 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:47.975652 22306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:47.975780 22306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "7369e1e5a20e493b9a148b6e8512dddc"
format_stamp: "Formatted at 2026-09-27 00:48:47 on dist-test-slave-l5sb"
server_key: "8e81f00647129ec24bdf5b357fcc423e"
server_key_iv: "0c830b81995fb13427d52ad0e94ac9ea"
server_key_version: "encryptionkey@0"
I20260927 00:48:47.975924 22306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:47.990280 22306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:47.993012 22306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:47.993216 22306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:48.000299 22306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34943
I20260927 00:48:48.000319 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34943 every 8 connection(s)
I20260927 00:48:48.001236 22306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:48.003931 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:48.006975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22306
I20260927 00:48:48.007082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:48.007354 392 external_mini_cluster.cc:1428] Setting key a4abda2c6d38b4e861f5711f55e66814
I20260927 00:48:48.009809 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc: Bootstrap starting.
I20260927 00:48:48.011919 22374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:48.012650 22374 log.cc:824] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:48.014549 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc: No bootstrap required, opened a new log
Sep 27 00:48:48 dist-test-slave-l5sb krb5kdc[22293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:48.018515 22374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } }
I20260927 00:48:48.018803 22374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:48.018862 22374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7369e1e5a20e493b9a148b6e8512dddc, State: Initialized, Role: FOLLOWER
I20260927 00:48:48.019354 22374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } }
I20260927 00:48:48.019524 22374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:48.019613 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:48.019709 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:48.020821 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } }
I20260927 00:48:48.021179 22374 leader_election.cc:302] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7369e1e5a20e493b9a148b6e8512dddc; no voters:
I20260927 00:48:48.021486 22374 leader_election.cc:288] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:48.021601 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:48.021840 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [term 1 LEADER]: Becoming Leader. State: Replica: 7369e1e5a20e493b9a148b6e8512dddc, State: Running, Role: LEADER
I20260927 00:48:48.021981 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:48.008727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35487 (local address 127.0.98.62:34943)
0927 00:48:48.009155 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:48.009170 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:48.009205 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:48:48.009886 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:48.009905 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:48.010276 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:48.010612 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:48.011422 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:48.012461 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:48.013374 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:48.013911 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:48.014072 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:48.016579 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:48.016585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:48.016601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:48.016632 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:48.018635 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:48.019204 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:48.019210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:48.019214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:48.019340 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:48.019672 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:48.019675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:48.019678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:48.020150 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:48.020321 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:48.020628 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 00:48:48.020951 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260927 00:48:48.022145 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } }
I20260927 00:48:48.022725 22374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:48.023272 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7369e1e5a20e493b9a148b6e8512dddc. Latest consensus state: current_term: 1 leader_uuid: "7369e1e5a20e493b9a148b6e8512dddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } } }
I20260927 00:48:48.023422 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:48.023764 22385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:48.024571 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7369e1e5a20e493b9a148b6e8512dddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7369e1e5a20e493b9a148b6e8512dddc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34943 } } }
I20260927 00:48:48.024670 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:48.024997 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:48.027944 22385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:48.033811 22385 catalog_manager.cc:1347] Generated new cluster ID: 2836c45361434ca2b6a67df577ec4509
I20260927 00:48:48.033905 22385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:48.042161 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:48.027269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50389 (local address 127.0.98.62:34943)
0927 00:48:48.027379 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:48.027383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:48.030021 (+ 2638us) server_negotiation.cc:407] Connection header received
0927 00:48:48.030205 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:48.030208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:48.030260 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:48.030346 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:48.031138 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:48.031816 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:48.032870 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:48.033119 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:48.033233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:48.038793 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:48.038797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:48.038800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:48.038826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:48.040490 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:48.040948 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:48.040952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:48.040954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:48.041037 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:48.041307 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:48.041310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:48.041312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:48.041523 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:48.041650 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:48.041891 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 00:48:48.042006 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 00:48:48.046028 22385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:48.046952 22385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:48.050414 22385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7369e1e5a20e493b9a148b6e8512dddc: Generated new TSK 0
I20260927 00:48:48.051162 22385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:48.059427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22306
2026-09-27T00:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:48 dist-test-slave-l5sb krb5kdc[22405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:48 dist-test-slave-l5sb krb5kdc[22405](info): set up 2 sockets
Sep 27 00:48:48 dist-test-slave-l5sb krb5kdc[22405](info): commencing operation
krb5kdc: starting...
W20260927 00:48:49.094086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:49 dist-test-slave-l5sb krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:49.217705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:49.323611 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:49.323935 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:49.324018 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:49.327939 22418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:49.328030 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:49.328069 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:49.328126 22418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:49.328166 22418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:49.333185 22418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:49.334201 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:49.335167 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:49.343577 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:49.343677 22418 server_base.cc:1034] running on GCE node
W20260927 00:48:49.343577 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:49.343616 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:49.344312 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:49.345409 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:49.346632 22418 hybrid_clock.cc:648] HybridClock initialized: now 1790470129346612 us; error 26 us; skew 500 ppm
Sep 27 00:48:49 dist-test-slave-l5sb krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:49.349444 22418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:49.350440 22418 webserver.cc:460] Webserver started at http://127.0.98.62:35623/ using document root <none> and password file <none>
I20260927 00:48:49.351007 22418 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:49.351099 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:49.351331 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:49.353394 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "fb8c4b660ce5470d87ab2e6a66beab2c"
format_stamp: "Formatted at 2026-09-27 00:48:49 on dist-test-slave-l5sb"
server_key: "23747360ce4b444b4b7a3752aa5ba244"
server_key_iv: "ee76e236ae38cf990be8049f97b2a85f"
server_key_version: "encryptionkey@0"
I20260927 00:48:49.354031 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb8c4b660ce5470d87ab2e6a66beab2c"
format_stamp: "Formatted at 2026-09-27 00:48:49 on dist-test-slave-l5sb"
server_key: "23747360ce4b444b4b7a3752aa5ba244"
server_key_iv: "ee76e236ae38cf990be8049f97b2a85f"
server_key_version: "encryptionkey@0"
I20260927 00:48:49.357950 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:48:49.360464 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:49.361724 22418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:49.361945 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "fb8c4b660ce5470d87ab2e6a66beab2c"
format_stamp: "Formatted at 2026-09-27 00:48:49 on dist-test-slave-l5sb"
server_key: "23747360ce4b444b4b7a3752aa5ba244"
server_key_iv: "ee76e236ae38cf990be8049f97b2a85f"
server_key_version: "encryptionkey@0"
I20260927 00:48:49.362072 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:49.378818 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:49.382200 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:49.382405 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:49.389258 22418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33773
I20260927 00:48:49.389282 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33773 every 8 connection(s)
I20260927 00:48:49.390249 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:49.392899 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:49.393280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22418
I20260927 00:48:49.393431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:49.393715 392 external_mini_cluster.cc:1428] Setting key 095e594ae4616e6161501d788071886e
I20260927 00:48:49.399506 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: Bootstrap starting.
I20260927 00:48:49.401613 22486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:49.402447 22486 log.cc:824] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:49.402695 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:49.395117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34867 (local address 127.0.98.62:33773)
0927 00:48:49.395527 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:49.395545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:49.395578 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:48:49.396274 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:49.396290 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:49.396626 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:49.396968 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:49.397361 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:49.398579 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:49.399504 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:49.400025 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:49.400189 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:49.401069 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:49.401074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:49.401086 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:49.401458 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:49.401510 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:49.401532 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:49.401637 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:49.401751 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 00:48:49.401982 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260927 00:48:49.404285 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: No bootstrap required, opened a new log
I20260927 00:48:49.405308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:49.408938 22486 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:48:49.409245 22486 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:49.409328 22486 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb8c4b660ce5470d87ab2e6a66beab2c, State: Initialized, Role: FOLLOWER
I20260927 00:48:49.409829 22486 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:48:49.410009 22486 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:49.410073 22486 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:49.410164 22486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:49.411154 22486 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:48:49.411541 22486 leader_election.cc:302] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb8c4b660ce5470d87ab2e6a66beab2c; no voters:
I20260927 00:48:49.411676 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:49.407568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36905 (local address 127.0.98.62:33773)
0927 00:48:49.407689 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:49.407693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:49.407803 (+ 110us) server_negotiation.cc:407] Connection header received
0927 00:48:49.407887 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:49.407890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:49.407933 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:49.408024 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:49.408242 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:49.408977 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:49.409942 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:49.410338 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:49.410452 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:49.411263 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:49.411267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:49.411269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:49.411282 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:49.411309 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:49.411321 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:49.411398 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:49.411493 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:48:49.411542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:48:49.411890 22486 leader_election.cc:288] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:49.412499 22496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:49.412689 22496 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [term 1 LEADER]: Becoming Leader. State: Replica: fb8c4b660ce5470d87ab2e6a66beab2c, State: Running, Role: LEADER
I20260927 00:48:49.412964 22496 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:48:49.413193 22486 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:49.414788 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb8c4b660ce5470d87ab2e6a66beab2c. Latest consensus state: current_term: 1 leader_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } } }
I20260927 00:48:49.414913 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:49.414839 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8c4b660ce5470d87ab2e6a66beab2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } } }
I20260927 00:48:49.414969 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c [sys.catalog]: This master's current role is: LEADER
W20260927 00:48:49.418267 22511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:49.418359 22511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:49.418452 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:49.419181 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:49.424472 22509 catalog_manager.cc:1347] Generated new cluster ID: 9d9b34070cae4b77b478ddf41c79e0fd
I20260927 00:48:49.424532 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:49.440690 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:49.441942 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:49.449137 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb8c4b660ce5470d87ab2e6a66beab2c: Generated new TSK 0
I20260927 00:48:49.449937 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:49.463702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22418
2026-09-27T00:48:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:49 dist-test-slave-l5sb krb5kdc[22517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:49 dist-test-slave-l5sb krb5kdc[22517](info): set up 2 sockets
Sep 27 00:48:49 dist-test-slave-l5sb krb5kdc[22517](info): commencing operation
krb5kdc: starting...
W20260927 00:48:50.578112 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:50 dist-test-slave-l5sb krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:50.701936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:50.819585 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:50.819895 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:50.819938 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:50.824020 22530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:50.824091 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:50.824111 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:50.824128 22530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:50.824144 22530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:50.829676 22530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:50.830842 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:50.831904 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:50.840516 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:50.840509 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:50.840518 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:50.840729 22530 server_base.cc:1034] running on GCE node
I20260927 00:48:50.841323 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:50.842555 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:50.843724 22530 hybrid_clock.cc:648] HybridClock initialized: now 1790470130843694 us; error 41 us; skew 500 ppm
Sep 27 00:48:50 dist-test-slave-l5sb krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:50.846668 22530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:50.847659 22530 webserver.cc:460] Webserver started at http://127.0.98.62:44511/ using document root <none> and password file <none>
I20260927 00:48:50.848259 22530 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:50.848326 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:50.848570 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:50.850854 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "b104c1a06a604d96a13af38c7a78c18c"
format_stamp: "Formatted at 2026-09-27 00:48:50 on dist-test-slave-l5sb"
server_key: "195b1991e8f44828f7fc03b62ba1ff87"
server_key_iv: "f8f62f1c8f45ca711f6aebaa474b241b"
server_key_version: "encryptionkey@0"
I20260927 00:48:50.851455 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "b104c1a06a604d96a13af38c7a78c18c"
format_stamp: "Formatted at 2026-09-27 00:48:50 on dist-test-slave-l5sb"
server_key: "195b1991e8f44828f7fc03b62ba1ff87"
server_key_iv: "f8f62f1c8f45ca711f6aebaa474b241b"
server_key_version: "encryptionkey@0"
I20260927 00:48:50.855358 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:48:50.858060 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:50.859339 22530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:50.859488 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "b104c1a06a604d96a13af38c7a78c18c"
format_stamp: "Formatted at 2026-09-27 00:48:50 on dist-test-slave-l5sb"
server_key: "195b1991e8f44828f7fc03b62ba1ff87"
server_key_iv: "f8f62f1c8f45ca711f6aebaa474b241b"
server_key_version: "encryptionkey@0"
I20260927 00:48:50.859615 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:50.864532 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:50.867704 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:50.867921 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:50.875463 22530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40111
I20260927 00:48:50.875494 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40111 every 8 connection(s)
I20260927 00:48:50.876531 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:50.878494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22530
I20260927 00:48:50.878619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:50.878914 392 external_mini_cluster.cc:1428] Setting key 337133bbc2de6202ddd6299c018bd5ad
I20260927 00:48:50.879786 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:50.884362 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:50.880381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47261 (local address 127.0.98.62:40111)
0927 00:48:50.880821 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:50.880848 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:50.880886 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:48:50.881547 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:50.881563 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:50.881936 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:50.882285 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:50.883148 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:50.883153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:50.883165 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:50.883176 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:50.883232 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:50.883253 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:50.883415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:50.883491 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 00:48:50.883650 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260927 00:48:50.886049 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c: Bootstrap starting.
I20260927 00:48:50.887290 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:50.888748 22598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:50.889631 22598 log.cc:824] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:50.891345 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:50.889563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53901 (local address 127.0.98.62:40111)
0927 00:48:50.889689 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:50.889693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:50.889872 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:48:50.889984 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:50.889988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:50.890021 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:50.890114 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:50.890969 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:50.890973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:50.890975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:48:50.890987 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:50.891014 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:48:50.891027 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:50.891118 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:50.891184 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 00:48:50.891222 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:48:50.891932 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c: No bootstrap required, opened a new log
I20260927 00:48:50.895411 22598 raft_consensus.cc:348] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 00:48:50.895659 22598 raft_consensus.cc:374] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:50.895756 22598 raft_consensus.cc:729] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b104c1a06a604d96a13af38c7a78c18c, State: Initialized, Role: FOLLOWER
I20260927 00:48:50.896296 22598 consensus_queue.cc:260] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 00:48:50.896428 22598 raft_consensus.cc:388] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:50.896471 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:50.896543 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:50.897377 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 00:48:50.897722 22598 leader_election.cc:302] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b104c1a06a604d96a13af38c7a78c18c; no voters:
I20260927 00:48:50.898058 22598 leader_election.cc:288] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:50.898191 22608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:50.898429 22608 raft_consensus.cc:686] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [term 1 LEADER]: Becoming Leader. State: Replica: b104c1a06a604d96a13af38c7a78c18c, State: Running, Role: LEADER
I20260927 00:48:50.898838 22608 consensus_queue.cc:237] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 00:48:50.899493 22598 sys_catalog.cc:564] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:50.900564 22609 sys_catalog.cc:455] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b104c1a06a604d96a13af38c7a78c18c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260927 00:48:50.900696 22609 sys_catalog.cc:458] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:50.901069 22617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:50.901015 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b104c1a06a604d96a13af38c7a78c18c. Latest consensus state: current_term: 1 leader_uuid: "b104c1a06a604d96a13af38c7a78c18c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b104c1a06a604d96a13af38c7a78c18c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260927 00:48:50.901335 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:50.903888 22617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:50.909143 22617 catalog_manager.cc:1347] Generated new cluster ID: 0dbc43909e404a5aa42ed0596e3f848f
I20260927 00:48:50.909211 22617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:50.928491 22617 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:50.928864 22617 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:50.937745 22617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b104c1a06a604d96a13af38c7a78c18c: Generated new TSK 0
I20260927 00:48:50.938618 22617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:48:50.944844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22530
2026-09-27T00:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1480 ms)
[----------] 18 tests from AuthTokenIssuingTest (25461 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:50 dist-test-slave-l5sb krb5kdc[22629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:50 dist-test-slave-l5sb krb5kdc[22629](info): set up 2 sockets
Sep 27 00:48:50 dist-test-slave-l5sb krb5kdc[22629](info): commencing operation
krb5kdc: starting...
W20260927 00:48:51.998211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:52.121263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:52.231094 22642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:52.231448 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:52.231556 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:52.231683 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:52.236549 22642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:52.236616 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:52.236637 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:52.236656 22642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:52.236720 22642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:52.242461 22642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:52.243559 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:52.244668 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:52.253264 22642 server_base.cc:1034] running on GCE node
W20260927 00:48:52.253312 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:52.253244 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:52.253182 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:52.254140 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:52.255214 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:52.256378 22642 hybrid_clock.cc:648] HybridClock initialized: now 1790470132256359 us; error 31 us; skew 500 ppm
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:52.264261 22642 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:52.265210 22642 webserver.cc:460] Webserver started at http://127.0.98.62:43575/ using document root <none> and password file <none>
I20260927 00:48:52.265846 22642 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:52.265920 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:52.266155 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:52.268159 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "4a146145a1e04bc694622f396f0c58b9"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "55bf2fdf128b10fd1cb027b163d79628"
server_key_iv: "dc9f7b3952baf7f509fbc060abc1e00b"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.268709 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a146145a1e04bc694622f396f0c58b9"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "55bf2fdf128b10fd1cb027b163d79628"
server_key_iv: "dc9f7b3952baf7f509fbc060abc1e00b"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.272313 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:52.274718 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:52.275961 22642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:52.276099 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "4a146145a1e04bc694622f396f0c58b9"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "55bf2fdf128b10fd1cb027b163d79628"
server_key_iv: "dc9f7b3952baf7f509fbc060abc1e00b"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.276209 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:52.304292 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:52.307582 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:52.307816 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:52.315055 22642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43105
I20260927 00:48:52.315371 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43105 every 8 connection(s)
I20260927 00:48:52.316090 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:52.317323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22642
I20260927 00:48:52.317461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:52.317842 392 external_mini_cluster.cc:1428] Setting key 7f9505f538a13ad7369a0d9b49fdbc02
I20260927 00:48:52.319455 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:52.323256 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:52.339948 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.325467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.98.62:43105)
0927 00:48:52.325931 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.325946 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.326126 (+ 180us) server_negotiation.cc:407] Connection header received
0927 00:48:52.327224 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.327229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.327410 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.327566 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.328698 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.330063 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.331646 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.332101 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.332221 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.334885 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.334891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.334907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.334926 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.336679 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.337268 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.337277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.337281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.337391 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.337719 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.337722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.337724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.338146 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.338287 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.338669 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.338919 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"spinlock_wait_cycles":59136,"thread_start_us":169,"threads_started":1}
I20260927 00:48:52.339949 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.325467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.0.1:34850)
0927 00:48:52.325939 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.325986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.326916 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.327619 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.327624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.328202 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.328539 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.328554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.330189 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.330194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.331517 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.331531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.332277 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.332282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.332434 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.333040 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.333067 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.334736 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.336908 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.336918 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.336927 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.337154 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.337489 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.337494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.337496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.337607 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.338280 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.338286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.338439 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.338609 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.338994 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":107,"thread_start_us":145,"threads_started":1}
I20260927 00:48:52.342545 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:52.343322 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.343061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45688)
0927 00:48:52.343174 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.343244 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:48:52.343530 22658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:48:52.344501 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.344273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45698)
0927 00:48:52.344381 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.344432 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:48:52.344655 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
I20260927 00:48:52.344990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37385
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:48:52.393921 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } attempt: 1
I20260927 00:48:52.394861 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.394554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45706)
0927 00:48:52.394719 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.394784 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 00:48:52.395560 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.395362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45720)
0927 00:48:52.395465 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.395512 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:48:52.395695 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
W20260927 00:48:52.453213 22717 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:52.453547 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:52.453622 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:52.453701 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:52.457701 22717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:52.457815 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:52.457859 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:52.457892 22717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:52.457922 22717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:52.463030 22717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:52.463860 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } attempt: 2
I20260927 00:48:52.464167 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:52.464655 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.464383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45734)
0927 00:48:52.464516 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.464584 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:48:52.465323 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:52.465361 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.465154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45746)
0927 00:48:52.465260 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.465306 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:48:52.465476 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37385: connect: Connection refused (error 111)
W20260927 00:48:52.473578 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:52.473603 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:52.473605 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:52.473881 22717 server_base.cc:1034] running on GCE node
I20260927 00:48:52.474457 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:52.475481 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:52.476675 22717 hybrid_clock.cc:648] HybridClock initialized: now 1790470132476666 us; error 25 us; skew 500 ppm
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:52.479419 22717 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:48:52.480338 22717 webserver.cc:460] Webserver started at http://127.0.98.61:34423/ using document root <none> and password file <none>
I20260927 00:48:52.480885 22717 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:52.480945 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:52.481103 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:52.483119 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data/instance:
uuid: "8b82337ddcc94b1ca2e98db4a5aec606"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "1a96ef7057203dd415d0f3061e8fe3d1"
server_key_iv: "c96e5b33487a459c8c37c196734eb8ca"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.483619 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal/instance:
uuid: "8b82337ddcc94b1ca2e98db4a5aec606"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "1a96ef7057203dd415d0f3061e8fe3d1"
server_key_iv: "c96e5b33487a459c8c37c196734eb8ca"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.487164 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:52.489408 22732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:52.490708 22717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:52.490862 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "8b82337ddcc94b1ca2e98db4a5aec606"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "1a96ef7057203dd415d0f3061e8fe3d1"
server_key_iv: "c96e5b33487a459c8c37c196734eb8ca"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.490986 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:52.509896 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:52.512547 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:52.512742 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:52.519447 22717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37385
I20260927 00:48:52.519711 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37385 every 8 connection(s)
I20260927 00:48:52.520424 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:48:52.520650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22717
I20260927 00:48:52.520748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-1/wal/instance
I20260927 00:48:52.521057 392 external_mini_cluster.cc:1428] Setting key 30bcc55a7d0a17fe3ffad92c34a5c9fb
I20260927 00:48:52.523272 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:52.527380 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:52.541604 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.529497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.0.98.62:43105)
0927 00:48:52.529636 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.529640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.530635 (+ 995us) server_negotiation.cc:407] Connection header received
0927 00:48:52.531562 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.531566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.531625 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.531792 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.533023 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.533649 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.535083 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.535509 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.535635 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.538050 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.538057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.538063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.538095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.539572 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.540091 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.540094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.540096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.540175 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.540516 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.540520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.540521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.540694 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.540806 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.541307 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.541431 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:48:52.542450 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.530003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.0.1:34858)
0927 00:48:52.530487 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.530527 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.531357 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.532008 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.532022 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.532333 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.532884 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.532897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.533851 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.533853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.534961 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.534970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.535653 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.535656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.535800 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.536451 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.536467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.537865 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.539707 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.539713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.539727 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.539983 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.540267 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.540279 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.540281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.540415 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.540808 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.540815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.541060 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.541239 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.541554 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
I20260927 00:48:52.543839 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:48:52.549611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38575
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:48:52.553632 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.544383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45762)
0927 00:48:52.544511 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.544522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.544580 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.545172 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.545177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.545323 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.545407 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.545412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.546550 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.546553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.547170 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.547179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.547780 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.547783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.547856 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.548252 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.548262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.549676 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.551774 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.551778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.551780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.552045 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.552361 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.552367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.552372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.552471 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.552938 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.552944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.553054 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.553172 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.553344 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:48:52.553637 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.544485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.0.98.61:37385)
0927 00:48:52.544832 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.544842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.544860 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:48:52.544893 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.544897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.545034 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.545172 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.545504 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.546460 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.547271 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.547681 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.547763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.549881 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.549893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.549908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.549937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.551664 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.552172 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.552177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.552178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.552258 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.552561 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.552563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.552564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.552831 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.552932 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.553246 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.553400 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":239,"thread_start_us":140,"threads_started":1}
I20260927 00:48:52.555608 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:52.556398 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.556168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58402)
0927 00:48:52.556286 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.556350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 00:48:52.556628 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } attempt: 3
W20260927 00:48:52.556672 22736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
I20260927 00:48:52.557682 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.557301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58404)
0927 00:48:52.557557 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.557611 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260927 00:48:52.557823 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:48:52.565367 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.557082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45778)
0927 00:48:52.557309 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.557320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.557382 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.557723 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.557726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.557951 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.558067 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.558073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.558810 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.558816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.559804 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.559813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.560285 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.560289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.560394 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.560835 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.560846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.562018 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.563793 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.563796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.563798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.563983 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.564271 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.564274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.564276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.564388 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.564859 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.564863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.564963 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.565071 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.565199 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 00:48:52.565450 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.557441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.0.98.61:37385)
0927 00:48:52.557551 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.557554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.557564 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:48:52.557589 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.557591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.557620 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.557705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.558180 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.558691 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.559926 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.560159 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.560265 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.562155 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.562158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.562161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.562185 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.563661 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.564071 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.564075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.564077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.564157 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.564506 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.564510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.564512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.564753 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.564873 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.565157 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.565267 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:48:52.566679 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:52.567366 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.567137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58414)
0927 00:48:52.567263 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.567310 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:48:52.568063 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.567808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58416)
0927 00:48:52.567969 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.568013 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 00:48:52.568225 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
I20260927 00:48:52.599408 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 1
I20260927 00:48:52.600251 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.599957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58428)
0927 00:48:52.600107 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.600173 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:48:52.600982 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.600779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58434)
0927 00:48:52.600886 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.600929 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:48:52.601125 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
I20260927 00:48:52.607056 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 1
I20260927 00:48:52.607780 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.607521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58448)
0927 00:48:52.607645 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.607693 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:48:52.608424 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.608220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58462)
0927 00:48:52.608326 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.608374 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:48:52.608542 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
W20260927 00:48:52.662345 22791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:52.662657 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:52.662727 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:52.662810 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:52.666754 22791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:52.666841 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:52.666880 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:52.666913 22791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:52.666941 22791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:52.672011 22791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43105,127.0.98.61:37385,127.0.98.60:38575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:52.673005 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:52.674018 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:52.676281 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 2
I20260927 00:48:52.676687 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 2
I20260927 00:48:52.677148 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.676808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58476)
0927 00:48:52.676984 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.677052 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:48:52.677381 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.677090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58482)
0927 00:48:52.677238 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.677296 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:48:52.677946 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.677653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58494)
0927 00:48:52.677822 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.677873 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 00:48:52.678126 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.677906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58508)
0927 00:48:52.678013 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.678061 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:48:52.678125 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
W20260927 00:48:52.678282 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38575: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38575: connect: Connection refused (error 111)
W20260927 00:48:52.682997 22798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:52.683089 22797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:52.683077 22791 server_base.cc:1034] running on GCE node
W20260927 00:48:52.683003 22800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:52.683686 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:52.684778 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:52.685961 22791 hybrid_clock.cc:648] HybridClock initialized: now 1790470132685944 us; error 25 us; skew 500 ppm
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:52.688689 22791 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:48:52.689486 22791 webserver.cc:460] Webserver started at http://127.0.98.60:38603/ using document root <none> and password file <none>
I20260927 00:48:52.690057 22791 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:52.690119 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:52.690279 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:52.692225 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data/instance:
uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "cca77dd15db0930cec178bedf3d42244"
server_key_iv: "a92747646bc7aae08b4fc61bec6529a9"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.692700 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal/instance:
uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "cca77dd15db0930cec178bedf3d42244"
server_key_iv: "a92747646bc7aae08b4fc61bec6529a9"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.696306 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:48:52.698521 22807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:52.699662 22791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:48:52.699791 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba"
format_stamp: "Formatted at 2026-09-27 00:48:52 on dist-test-slave-l5sb"
server_key: "cca77dd15db0930cec178bedf3d42244"
server_key_iv: "a92747646bc7aae08b4fc61bec6529a9"
server_key_version: "encryptionkey@0"
I20260927 00:48:52.699891 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:52.729348 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:52.732097 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:52.732293 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:52.740234 22791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38575
I20260927 00:48:52.740505 22859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38575 every 8 connection(s)
I20260927 00:48:52.741195 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:48:52.743870 22860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:52.746308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 22791
I20260927 00:48:52.746440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469854491772-392-0/minicluster-data/master-2/wal/instance
I20260927 00:48:52.746716 392 external_mini_cluster.cc:1428] Setting key e68d57fb779ab926c63da1c7d9fe086e
I20260927 00:48:52.747905 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:48:52.759106 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.749617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.98.62:43105)
0927 00:48:52.749764 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.749785 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.750114 (+ 329us) server_negotiation.cc:407] Connection header received
0927 00:48:52.750355 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.750358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.750411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.750511 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.751366 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.751843 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.752816 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.753013 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.753090 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.755601 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.755607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.755611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.755636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.757033 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.757584 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.757588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.757590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.757671 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.758042 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.758046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.758047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.758230 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.758338 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.758828 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.758940 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:48:52.759898 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.749485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.0.1:34862)
0927 00:48:52.750001 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.750046 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.750189 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.750553 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.750557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.751017 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.751203 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.751216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.751958 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.751962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.752694 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.752702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.753147 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.753155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.753323 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.754052 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.754067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.755460 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.757146 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.757156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.757164 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.757482 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.757791 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.757796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.757797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.757937 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.758347 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.758356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.758578 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.758750 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.759030 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":69,"spinlock_wait_cycles":2432,"thread_start_us":120,"threads_started":1}
I20260927 00:48:52.759887 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.748055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.98.60:38575)
0927 00:48:52.748427 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.748452 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.748488 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:52.749119 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.749136 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.749491 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.749876 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.750663 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.751844 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.752669 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.753104 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.753248 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.756152 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.756158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.756167 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.756187 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.757938 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.758360 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.758366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.758367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.758442 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.758709 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.758714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.758716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.758968 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.759481 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.759590 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.759692 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":241,"thread_start_us":145,"threads_started":1}
I20260927 00:48:52.761677 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:52.769464 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 3
I20260927 00:48:52.769994 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.762152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37385 (local address 127.0.0.1:45786)
0927 00:48:52.762259 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.762271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.762342 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.762727 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.762730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.762913 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.762996 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.763001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.763709 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.763713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.764267 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.764276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.764696 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.764699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.764801 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.765225 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.765236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.766641 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.768514 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.768518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.768519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.768745 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.769007 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.769012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.769015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.769102 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.769451 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.769453 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.769521 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.769673 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.769843 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 00:48:52.770004 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.762413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.0.98.61:37385)
0927 00:48:52.762535 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.762539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.762549 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:48:52.762584 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.762586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.762622 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.762715 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.763112 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.763593 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.764385 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.764591 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.764697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.766764 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.766767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.766769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.766792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.768405 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.768844 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.768847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.768848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.768916 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.769194 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.769196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.769197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.769355 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.769453 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.769733 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.769870 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:52.770998 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:52.771679 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.762045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.62:43105)
0927 00:48:52.762150 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.762154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.762231 (+ 77us) server_negotiation.cc:407] Connection header received
0927 00:48:52.762506 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.762508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.762541 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.762629 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.763196 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.763656 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.764494 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.764716 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.764823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.767428 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.767432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.767434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.767457 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.769107 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.769910 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.769913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.769914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.769999 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.770318 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.770321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.770323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.770534 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.770735 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.771356 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.771478 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:48:52.777251 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } attempt: 3
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:48:52.782044 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.769858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58532)
0927 00:48:52.770092 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.770106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.770171 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.770562 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.770564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.770742 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.770870 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.770877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.772066 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.772070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.772848 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.772860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.773565 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.773581 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.773660 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.774126 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.774139 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.775532 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.778651 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.778654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.778656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.778894 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.779356 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.779362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.779367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.779499 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.779948 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.779951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.780037 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.781702 (+ 1665us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.781866 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 00:48:52.782286 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.770303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.98.60:38575)
0927 00:48:52.770394 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.770397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.770406 (+ 9us) server_negotiation.cc:407] Connection header received
0927 00:48:52.770431 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.770432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.770460 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.770545 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.770964 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.771958 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.772977 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.773377 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.773466 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.775651 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.775654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.775656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.775697 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.778524 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.779032 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.779035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.779037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.779169 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.779605 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.779610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.779615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.779817 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.779902 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.781753 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.782135 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 00:48:52.783988 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.771365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58542)
0927 00:48:52.771568 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.771580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.771650 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.772476 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.772479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.772626 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.772687 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.772692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.774123 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.774126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.774899 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.774924 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.775605 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.775611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.776474 (+ 863us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.776919 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.776933 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.779278 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.782451 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.782453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.782455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.782677 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.782975 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.782978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.782980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.783067 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.783538 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.783541 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.783617 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.783714 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.783831 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 00:48:52.784791 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.771541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.0.98.60:38575)
0927 00:48:52.772187 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.772195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.772206 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:48:52.772246 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.772248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.772300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.772394 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.772797 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.774007 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.775043 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.775417 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.775511 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.779450 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.779478 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.779481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.779502 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.782338 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.782781 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.782784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.782786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.782879 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.783162 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.783165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.783167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.783412 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.783505 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.784509 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.784613 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":88,"threads_started":1}
Sep 27 00:48:52 dist-test-slave-l5sb krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:48:52.785497 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.772956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.0.98.61:37385)
0927 00:48:52.773062 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.773068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.773228 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:48:52.773403 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.773407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.773456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.773510 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.774597 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.775380 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.776578 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.776941 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.777044 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.780478 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.780482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.780485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.780513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.783565 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.784051 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.784056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.784059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.784153 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.784468 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.784474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.784478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.784666 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.784880 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.785235 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.785340 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:48:52.787237 22860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: Bootstrap starting.
I20260927 00:48:52.788501 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.777595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38575 (local address 127.0.0.1:58552)
0927 00:48:52.777795 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:52.777820 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:52.777896 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:52.780518 (+ 2622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:52.780520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:52.780664 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:52.780792 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.780799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.782037 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.782040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.782791 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:52.782802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.783345 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.783349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:52.783431 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.783851 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:52.783870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:52.785163 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:52.787078 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.787081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.787083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.787275 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.787562 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:52.787565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:52.787567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:52.787649 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:52.788009 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:52.788012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:52.788108 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:48:52.788209 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:48:52.788349 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 00:48:52.788514 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.778366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.98.60:38575)
0927 00:48:52.780155 (+ 1789us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.780180 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.780196 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:48:52.780254 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.780257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.780334 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.780912 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.781011 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.781888 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.782925 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.783216 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.783639 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.785306 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.785309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.785312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.785335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.786942 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.787374 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.787379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.787381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.787460 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.787745 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.787748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.787749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.787913 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.788016 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.788268 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.788366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":88,"threads_started":1}
I20260927 00:48:52.789595 22860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:52.786084 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: Bootstrap starting.
I20260927 00:48:52.790470 22860 log.cc:824] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:52.791229 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: Bootstrap starting.
I20260927 00:48:52.792228 22860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: No bootstrap required, opened a new log
I20260927 00:48:52.792346 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:52.793131 22785 log.cc:824] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:52.793530 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:52.795329 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: No bootstrap required, opened a new log
I20260927 00:48:52.796120 22860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.796345 22860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57188eb1c3c3479d8eb0b92ddb9df0ba, State: Initialized, Role: FOLLOWER
I20260927 00:48:52.796635 22860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.797041 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:52.797075 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.797338 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.797914 22860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:52.798521 22710 log.cc:824] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:52.799384 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.799597 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b82337ddcc94b1ca2e98db4a5aec606, State: Initialized, Role: FOLLOWER
I20260927 00:48:52.798377 22684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4a146145a1e04bc694622f396f0c58b9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34874
I20260927 00:48:52.799897 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.800755 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.801064 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.801189 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:52.802088 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: No bootstrap required, opened a new log
I20260927 00:48:52.801620 22791 master_runner.cc:186] Error getting master registration for 127.0.98.62:43105: OK, instance_id { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" instance_seqno: 1790470132312967 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:48:52.804968 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.805150 22791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:48:52.805178 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a146145a1e04bc694622f396f0c58b9, State: Initialized, Role: FOLLOWER
I20260927 00:48:52.805476 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.806787 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:52.806880 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:48:52.806990 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:48:52.807587 22881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:52.807655 22881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:52.808216 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
I20260927 00:48:52.804503 22717 master_runner.cc:186] Error getting master registration for 127.0.98.62:43105: OK, instance_id { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" instance_seqno: 1790470132312967 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:48:52.808722 22717 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:48:52.809166 22684 leader_election.cc:288] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b82337ddcc94b1ca2e98db4a5aec606 (127.0.98.61:37385), 57188eb1c3c3479d8eb0b92ddb9df0ba (127.0.98.60:38575)
W20260927 00:48:52.809394 22663 tablet.cc:2369] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:52.809551 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.809644 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.809681 22759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a146145a1e04bc694622f396f0c58b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b82337ddcc94b1ca2e98db4a5aec606"
I20260927 00:48:52.809935 22759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:52.810397 22834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a146145a1e04bc694622f396f0c58b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba"
I20260927 00:48:52.810602 22834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:52.811301 22759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a146145a1e04bc694622f396f0c58b9 in term 1.
I20260927 00:48:52.811715 22658 leader_election.cc:302] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a146145a1e04bc694622f396f0c58b9, 8b82337ddcc94b1ca2e98db4a5aec606; no voters:
I20260927 00:48:52.811959 22834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a146145a1e04bc694622f396f0c58b9 in term 1.
I20260927 00:48:52.812016 22894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:52.812194 22894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a146145a1e04bc694622f396f0c58b9, State: Running, Role: LEADER
W20260927 00:48:52.812507 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:48:52.812505 22894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } }
W20260927 00:48:52.812623 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:52.815426 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a146145a1e04bc694622f396f0c58b9. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.815541 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:52.815934 22918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 00:48:52.816085 22893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:52.816190 22893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:52.816846 22918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:52.823859 22834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [term 1 FOLLOWER]: Refusing update from remote peer 4a146145a1e04bc694622f396f0c58b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:52.824509 22909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:52.827142 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.814087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.98.61:37385)
0927 00:48:52.814196 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.814201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.815998 (+ 1797us) server_negotiation.cc:407] Connection header received
0927 00:48:52.817515 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.817518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.817567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.817616 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.818629 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.819341 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.820491 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.820830 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.820936 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.822891 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.822895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.822898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.822924 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.824659 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.825139 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.825143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.825144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.825236 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.825545 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.825548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.825550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.825804 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.825953 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.826872 (+ 919us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.826986 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:48:52.828799 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a146145a1e04bc694622f396f0c58b9. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.828920 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.829821 22759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [term 1 FOLLOWER]: Refusing update from remote peer 4a146145a1e04bc694622f396f0c58b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:52.830432 22894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:52.836472 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a146145a1e04bc694622f396f0c58b9. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.836601 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.836905 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.837045 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.837316 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.814564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.98.62:43105)
0927 00:48:52.814678 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.814683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.826694 (+ 12011us) server_negotiation.cc:407] Connection header received
0927 00:48:52.826802 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.826815 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.826863 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.826946 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.828098 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.828882 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.830072 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.830377 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.830461 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.832203 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.832206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.832209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.832237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.834234 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.834898 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.834902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.834904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.834987 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.835433 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.835437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.835439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.835641 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.835767 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.836931 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.837146 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:48:52.838131 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.838241 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:52.838603 22919 mvcc.cc:205] Tried to move back new op lower bound from 7333765664039333888 to 7333765664003612672. Current Snapshot: MvccSnapshot[applied={T|T < 7333765664039333888}]
I20260927 00:48:52.840471 22918 catalog_manager.cc:1347] Generated new cluster ID: 8e48e7f52dfc4e02b25c536817bc49d9
I20260927 00:48:52.840524 22918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:52.847579 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:52.814958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.0.98.60:38575)
0927 00:48:52.815069 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.815074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.838114 (+ 23040us) server_negotiation.cc:407] Connection header received
0927 00:48:52.838159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.838162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.838213 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.838288 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.840097 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.840836 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.842003 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:52.842364 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:52.842445 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:52.844063 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:52.844067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:52.844070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:52.844095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:52.845666 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.846130 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.846133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.846135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.846206 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:52.846454 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:52.846456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:52.846458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:52.846640 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:52.846758 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:52.847321 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 00:48:52.847426 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:48:52.860800 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.860968 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:52.862203 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a146145a1e04bc694622f396f0c58b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a146145a1e04bc694622f396f0c58b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } peers { permanent_uuid: "8b82337ddcc94b1ca2e98db4a5aec606" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37385 } } peers { permanent_uuid: "57188eb1c3c3479d8eb0b92ddb9df0ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38575 } } }
I20260927 00:48:52.862332 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:52.863981 22923 mvcc.cc:205] Tried to move back new op lower bound from 7333765664039333888 to 7333765664003612672. Current Snapshot: MvccSnapshot[applied={T|T < 7333765664039333888}]
I20260927 00:48:52.864387 22920 mvcc.cc:205] Tried to move back new op lower bound from 7333765664039333888 to 7333765664003612672. Current Snapshot: MvccSnapshot[applied={T|T < 7333765664039333888}]
I20260927 00:48:52.902390 22918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:52.904659 22918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:52.921672 22918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a146145a1e04bc694622f396f0c58b9: Generated new TSK 0
I20260927 00:48:52.922605 22918 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:48:52.964668 22714 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.962246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.98.62:43105)
0927 00:48:52.962373 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.962379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.963560 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 00:48:52.963792 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.963796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.963849 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.963903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.964524 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34892: BlockingRecv error: recv got EOF from 127.0.0.1:34892 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 00:48:52.966059 22866 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.962939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.0.98.60:38575)
0927 00:48:52.963046 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.963051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.965150 (+ 2099us) server_negotiation.cc:407] Connection header received
0927 00:48:52.965329 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.965332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.965379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.965428 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.965945 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58570: BlockingRecv error: recv got EOF from 127.0.0.1:58570 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 00:48:52.967913 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.966685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.0.98.61:37385)
0927 00:48:52.966845 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.966850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.967052 (+ 202us) server_negotiation.cc:407] Connection header received
0927 00:48:52.967209 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.967212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.967257 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.967317 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.967795 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45820: BlockingRecv error: recv got EOF from 127.0.0.1:45820 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:48:52.971753 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.970373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.0.98.61:37385)
0927 00:48:52.970466 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.970469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.970879 (+ 410us) server_negotiation.cc:407] Connection header received
0927 00:48:52.971081 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.971084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.971152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.971200 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.971663 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45836: BlockingRecv error: recv got EOF from 127.0.0.1:45836 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:48:52.976737 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.975208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.61:37385)
0927 00:48:52.975300 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.975315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.975851 (+ 536us) server_negotiation.cc:407] Connection header received
0927 00:48:52.976020 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.976023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.976068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.976115 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.976644 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45842: BlockingRecv error: recv got EOF from 127.0.0.1:45842 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:48:52.982277 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.980877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.0.98.61:37385)
0927 00:48:52.980974 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.980977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.981410 (+ 433us) server_negotiation.cc:407] Connection header received
0927 00:48:52.981574 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.981577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.981622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.981668 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.982191 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45844: BlockingRecv error: recv got EOF from 127.0.0.1:45844 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 00:48:52.990005 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:52.988332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.0.98.61:37385)
0927 00:48:52.988455 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:52.988460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:52.989004 (+ 544us) server_negotiation.cc:407] Connection header received
0927 00:48:52.989207 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:52.989210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:52.989258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:52.989319 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:52.989919 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45850: BlockingRecv error: recv got EOF from 127.0.0.1:45850 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 00:48:53.016532 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.000572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.0.98.61:37385)
0927 00:48:53.000687 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.000691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.015332 (+ 14641us) server_negotiation.cc:407] Connection header received
0927 00:48:53.015606 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.015610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.015688 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.015753 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.016403 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45858: BlockingRecv error: recv got EOF from 127.0.0.1:45858 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 00:48:53.036772 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.035295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.98.61:37385)
0927 00:48:53.035451 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.035456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.035769 (+ 313us) server_negotiation.cc:407] Connection header received
0927 00:48:53.035902 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.035905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.035956 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.036032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.036647 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45860: BlockingRecv error: recv got EOF from 127.0.0.1:45860 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:48:53.072539 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.071290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.0.98.61:37385)
0927 00:48:53.071446 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.071451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.071591 (+ 140us) server_negotiation.cc:407] Connection header received
0927 00:48:53.071806 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.071810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.071871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.071964 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.072420 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45872: BlockingRecv error: recv got EOF from 127.0.0.1:45872 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 00:48:53.139911 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.138639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.61:37385)
0927 00:48:53.138803 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.138808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.138944 (+ 136us) server_negotiation.cc:407] Connection header received
0927 00:48:53.139118 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.139122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.139184 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.139283 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.139751 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45876: BlockingRecv error: recv got EOF from 127.0.0.1:45876 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 00:48:53.271762 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.270252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.98.61:37385)
0927 00:48:53.270414 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.270419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.270524 (+ 105us) server_negotiation.cc:407] Connection header received
0927 00:48:53.270693 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.270696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.270772 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.270875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.271645 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45878: BlockingRecv error: recv got EOF from 127.0.0.1:45878 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 00:48:53.531198 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:53.530014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.98.61:37385)
0927 00:48:53.530177 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:53.530181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:53.530321 (+ 140us) server_negotiation.cc:407] Connection header received
0927 00:48:53.530462 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:53.530464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:53.530512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:53.530584 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:53.531082 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45894: BlockingRecv error: recv got EOF from 127.0.0.1:45894 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:48:53.809042 22881 catalog_manager.cc:1259] Loaded cluster ID: 8e48e7f52dfc4e02b25c536817bc49d9
I20260927 00:48:53.809131 22881 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: loading cluster ID for follower catalog manager: success
I20260927 00:48:53.810838 22881 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: acquiring CA information for follower catalog manager: success
I20260927 00:48:53.812011 22881 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57188eb1c3c3479d8eb0b92ddb9df0ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:48:53.817286 22893 catalog_manager.cc:1259] Loaded cluster ID: 8e48e7f52dfc4e02b25c536817bc49d9
I20260927 00:48:53.817350 22893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: loading cluster ID for follower catalog manager: success
I20260927 00:48:53.819024 22893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: acquiring CA information for follower catalog manager: success
I20260927 00:48:53.820117 22893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b82337ddcc94b1ca2e98db4a5aec606: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:48:54.049544 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:54.045535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.61:37385)
0927 00:48:54.045918 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:54.045922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:54.045937 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:48:54.045989 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:54.045991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:54.046115 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:54.046250 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:54.046509 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.047042 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.047954 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.048185 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.048273 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:54.048589 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:54.049066 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:54.049167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:54.049277 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 00:48:54.049327 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260927 00:48:54.056530 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:54.052982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.0.98.60:38575)
0927 00:48:54.053234 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:54.053237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:54.053249 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:48:54.053326 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:54.053328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:54.053439 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:54.053537 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:54.053736 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.054332 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.055128 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.055358 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.055439 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:54.055699 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:54.056119 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:54.056232 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:54.056283 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:48:54.056330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260927 00:48:54.064251 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:54.060243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.0.98.60:38575)
0927 00:48:54.060365 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:54.060369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:54.060467 (+ 98us) server_negotiation.cc:407] Connection header received
0927 00:48:54.060574 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:54.060577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:54.060631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:54.060703 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:54.061187 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.061981 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.063129 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.063349 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.063460 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:54.063714 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:54.063822 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:54.063889 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:54.064054 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 00:48:54.064101 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:48:54.064332 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:54.059629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.61:37385)
0927 00:48:54.059770 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:54.059775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:54.060139 (+ 364us) server_negotiation.cc:407] Connection header received
0927 00:48:54.060479 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:54.060483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:54.060555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:54.060606 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:54.061178 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.061807 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.063070 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.063410 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.063482 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:54.063833 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:54.063917 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:54.063990 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:54.064124 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:48:54.064183 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:48:54.065565 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:54.059577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.0.98.62:43105)
0927 00:48:54.059800 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:54.059806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:54.060096 (+ 290us) server_negotiation.cc:407] Connection header received
0927 00:48:54.060381 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:54.060384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:54.060565 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:54.060659 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:54.061104 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.061967 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.063873 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:54.064224 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:54.064316 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:54.064745 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:54.065111 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:54.065194 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:54.065344 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:48:54.065407 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260927 00:48:54.068513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22642
I20260927 00:48:54.075094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22717
I20260927 00:48:54.081290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 22791
2026-09-27T00:48:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3138 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:54 dist-test-slave-l5sb krb5kdc[23007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:54 dist-test-slave-l5sb krb5kdc[23007](info): set up 2 sockets
Sep 27 00:48:54 dist-test-slave-l5sb krb5kdc[23007](info): commencing operation
krb5kdc: starting...
W20260927 00:48:55.118161 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:55.241904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:55.353008 23020 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:55.353314 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:55.353339 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:55.353438 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:55.357585 23020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:55.357656 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:55.357676 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:55.357693 23020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:55.357709 23020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:55.363209 23020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:55.364323 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:55.365355 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:55.373941 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.373971 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.373971 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:55.374029 23020 server_base.cc:1034] running on GCE node
I20260927 00:48:55.374799 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:55.375885 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:55.377060 23020 hybrid_clock.cc:648] HybridClock initialized: now 1790470135377033 us; error 38 us; skew 500 ppm
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:55.379868 23020 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:55.380738 23020 webserver.cc:460] Webserver started at http://127.0.98.62:33805/ using document root <none> and password file <none>
I20260927 00:48:55.381309 23020 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:55.381404 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:55.381629 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:55.383692 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data/instance:
uuid: "b09b238b83f64b0ea056e40ce3e2df2d"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "7ffe5e54d41a61f76576dd8e6f901e4f"
server_key_iv: "2cd6ea39f6538f149e0045751e7fba96"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.384263 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance:
uuid: "b09b238b83f64b0ea056e40ce3e2df2d"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "7ffe5e54d41a61f76576dd8e6f901e4f"
server_key_iv: "2cd6ea39f6538f149e0045751e7fba96"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.387957 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:48:55.390436 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:55.391628 23020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:55.391773 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal
uuid: "b09b238b83f64b0ea056e40ce3e2df2d"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "7ffe5e54d41a61f76576dd8e6f901e4f"
server_key_iv: "2cd6ea39f6538f149e0045751e7fba96"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.391892 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:55.423228 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:55.426345 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:55.426563 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:55.433544 23020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45023
I20260927 00:48:55.433907 23087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45023 every 8 connection(s)
I20260927 00:48:55.434597 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:55.437311 23088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:55.438084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 23020
I20260927 00:48:55.438232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:55.438537 392 external_mini_cluster.cc:1428] Setting key 55d4747efe304bdd4f5cf7a445ba3465
I20260927 00:48:55.441656 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:55.457412 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.444116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.98.62:45023)
0927 00:48:55.444613 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.444634 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.444885 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:48:55.445915 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.445921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.446173 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.446392 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.447120 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.448056 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.449060 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.449488 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.449624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.452359 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.452369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.452386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.452417 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.454210 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.454762 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.454767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.454768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.454870 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.455231 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.455233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.455234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.455689 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.455794 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.456274 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.456530 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260927 00:48:55.457418 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.444116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.0.1:57926)
0927 00:48:55.444717 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.444763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.445562 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.446394 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.446399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.446708 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.446983 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.446996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.448172 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.448175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.448921 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.448930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.449644 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.449647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.449742 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.450489 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.450505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.452198 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.454310 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.454322 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.454336 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.454657 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.454962 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.454968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.454973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.455136 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.455820 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.455827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.456006 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.456221 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.456641 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":79,"spinlock_wait_cycles":96128,"thread_start_us":143,"threads_started":1}
I20260927 00:48:55.459916 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:55.460752 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.460476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:55996)
0927 00:48:55.460606 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.460691 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 00:48:55.460945 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
I20260927 00:48:55.461936 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.461662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56002)
0927 00:48:55.461763 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.461884 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:48:55.462146 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:48:55.469446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33177
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:48:55.511430 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 1
I20260927 00:48:55.512348 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.512039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56018)
0927 00:48:55.512194 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.512259 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:48:55.513046 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.512840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56024)
0927 00:48:55.512947 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.512995 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:48:55.513175 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
I20260927 00:48:55.581341 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 2
W20260927 00:48:55.581400 23095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:55.581837 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:55.581882 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:55.581944 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:48:55.582218 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.581932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56026)
0927 00:48:55.582084 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.582145 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 00:48:55.582923 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.582705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56040)
0927 00:48:55.582820 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.582873 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:48:55.583045 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
W20260927 00:48:55.586238 23095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:55.586318 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:55.586361 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:55.586403 23095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:55.586434 23095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:55.591702 23095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:55.592823 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:55.593890 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:55.602468 23095 server_base.cc:1034] running on GCE node
W20260927 00:48:55.602393 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.602422 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.602408 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:55.603150 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:55.604264 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:55.605459 23095 hybrid_clock.cc:648] HybridClock initialized: now 1790470135605427 us; error 44 us; skew 500 ppm
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:55.608273 23095 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:48:55.609163 23095 webserver.cc:460] Webserver started at http://127.0.98.61:45661/ using document root <none> and password file <none>
I20260927 00:48:55.609714 23095 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:55.609838 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:55.610065 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:55.612149 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data/instance:
uuid: "60fcfbf7fa134bc393678c04fc3af63c"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "12887134c49d17e05361b7e9dbb281d6"
server_key_iv: "17d3b1d3110fcf239af9b7ec9217bcb1"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.612696 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal/instance:
uuid: "60fcfbf7fa134bc393678c04fc3af63c"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "12887134c49d17e05361b7e9dbb281d6"
server_key_iv: "17d3b1d3110fcf239af9b7ec9217bcb1"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.616262 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:48:55.618635 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:55.619897 23095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:55.620030 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal
uuid: "60fcfbf7fa134bc393678c04fc3af63c"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "12887134c49d17e05361b7e9dbb281d6"
server_key_iv: "17d3b1d3110fcf239af9b7ec9217bcb1"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.620157 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:55.641448 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:55.644783 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:55.644992 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:55.651899 23095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33177
I20260927 00:48:55.652185 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33177 every 8 connection(s)
I20260927 00:48:55.652848 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:48:55.655404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 23095
I20260927 00:48:55.655551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-1/wal/instance
I20260927 00:48:55.655882 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:55.655889 392 external_mini_cluster.cc:1428] Setting key 38a25b1eeeb73dca794b9dc3f198abfc
I20260927 00:48:55.659653 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:55.673316 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.661528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.62:45023)
0927 00:48:55.661673 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.661677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.662725 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 00:48:55.663679 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.663682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.663743 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.663847 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.665011 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.665669 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.666927 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.667164 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.667277 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.669867 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.669871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.669873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.669896 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.671324 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.671861 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.671864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.671865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.671935 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.672259 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.672262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.672264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.672430 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.672534 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.673012 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.673144 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:48:55.674191 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.662062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.0.1:57932)
0927 00:48:55.662565 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.662605 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.663472 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.664036 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.664043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.664429 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.664871 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.664884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.665834 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.665837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.666802 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.666811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.667319 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.667323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.667480 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.668065 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.668080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.669665 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.671440 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.671446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.671459 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.671708 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.672024 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.672029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.672031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.672161 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.672532 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.672539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.672774 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.672945 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.673253 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":146,"threads_started":1}
I20260927 00:48:55.674199 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 3
I20260927 00:48:55.675662 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:48:55.682004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
/tmp/dist-test-taskjP59Ka/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41733
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:48:55.684648 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.674562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56044)
0927 00:48:55.674730 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.674745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.674845 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.675480 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.675483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.675639 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.675768 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.675784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.677036 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.677039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.677706 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.677716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.678551 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.678554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.678657 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.679150 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.679164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.680682 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.682983 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.682988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.682990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.683212 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.683533 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.683539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.683542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.683648 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.684131 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.684135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.684267 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.684381 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.684518 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 00:48:55.686347 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.674710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.0.98.61:33177)
0927 00:48:55.675049 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.675061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.675082 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:48:55.675123 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.675128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.675311 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.675441 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.675894 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.676921 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.677864 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.678436 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.678519 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.680830 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.680841 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.680854 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.680879 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.682881 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.683327 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.683336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.683337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.683432 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.683743 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.683748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.683750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.684024 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.684127 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.685970 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.686134 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260927 00:48:55.686475 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.676147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56060)
0927 00:48:55.676254 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.676268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.676364 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.676845 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.676848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.677045 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.677154 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.677161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.678157 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.678159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.678878 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.678907 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.679558 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.679562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.679723 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.680225 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.680240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.681689 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.684867 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.684874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.684879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.685129 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.685406 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.685412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.685417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.685516 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.685900 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.685903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.686008 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.686118 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.686251 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":51,"spinlock_wait_cycles":38144}
I20260927 00:48:55.686504 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.676194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.0.98.61:33177)
0927 00:48:55.676486 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.676495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.676507 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:48:55.676540 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.676542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.676586 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.676669 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.677296 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.678028 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.679044 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.679414 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.679541 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.681866 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.681869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.681872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.681895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.684642 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.685238 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.685241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.685242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.685313 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.685604 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.685606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.685607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.685762 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.685900 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.686162 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.686308 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260927 00:48:55.688493 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:55.688657 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:55.689332 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.689095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50550)
0927 00:48:55.689220 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.689269 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:48:55.689677 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.689395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50564)
0927 00:48:55.689540 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.689623 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 00:48:55.689985 23112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
I20260927 00:48:55.690222 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.689927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50574)
0927 00:48:55.690100 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.690144 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 00:48:55.690367 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
I20260927 00:48:55.690869 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.690664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50576)
0927 00:48:55.690779 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.690821 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:48:55.691035 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
I20260927 00:48:55.721517 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 1
I20260927 00:48:55.722329 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.722044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50588)
0927 00:48:55.722186 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.722252 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:48:55.723047 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.722846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50602)
0927 00:48:55.722949 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.722999 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:48:55.723181 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
I20260927 00:48:55.740257 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 1
I20260927 00:48:55.741032 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.740768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50612)
0927 00:48:55.740905 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.740961 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:48:55.741724 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.741517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50618)
0927 00:48:55.741624 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.741672 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:48:55.741922 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
W20260927 00:48:55.794049 23170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:48:55.794389 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:55.794454 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:55.794531 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:48:55.798382 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 2
W20260927 00:48:55.798996 23170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:55.799060 23170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:55.799090 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:55.799146 23170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:55.799172 23170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:55.799289 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.798912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50624)
0927 00:48:55.799103 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.799186 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 00:48:55.800029 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.799833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50630)
0927 00:48:55.799931 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.799977 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:48:55.800226 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
I20260927 00:48:55.804749 23170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45023,127.0.98.61:33177,127.0.98.60:41733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:48:55.805763 23170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:55.806833 23170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:48:55.810067 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 2
I20260927 00:48:55.810846 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.810541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50632)
0927 00:48:55.810693 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.810762 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:48:55.811611 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.811426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50646)
0927 00:48:55.811513 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.811561 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 00:48:55.811892 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41733: connect: Connection refused (error 111)
W20260927 00:48:55.815725 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.815721 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:55.815743 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:55.816022 23170 server_base.cc:1034] running on GCE node
I20260927 00:48:55.816610 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:55.817672 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:55.818874 23170 hybrid_clock.cc:648] HybridClock initialized: now 1790470135818853 us; error 30 us; skew 500 ppm
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:55.821982 23170 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:48:55.822878 23170 webserver.cc:460] Webserver started at http://127.0.98.60:34543/ using document root <none> and password file <none>
I20260927 00:48:55.823467 23170 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:55.823566 23170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:55.823830 23170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:48:55.826038 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data/instance:
uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "ff8add6d1e0a7d661fdbeecd969d5140"
server_key_iv: "5e7e7446fee0ef7e6008c4e962c22e04"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.826628 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal/instance:
uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "ff8add6d1e0a7d661fdbeecd969d5140"
server_key_iv: "5e7e7446fee0ef7e6008c4e962c22e04"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.830399 23170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:48:55.832859 23186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:55.834110 23170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:48:55.834250 23170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal
uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf"
format_stamp: "Formatted at 2026-09-27 00:48:55 on dist-test-slave-l5sb"
server_key: "ff8add6d1e0a7d661fdbeecd969d5140"
server_key_iv: "5e7e7446fee0ef7e6008c4e962c22e04"
server_key_version: "encryptionkey@0"
I20260927 00:48:55.834367 23170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:55.870566 23170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:55.874253 23170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:55.874478 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:55.881850 23170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41733
I20260927 00:48:55.882138 23238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41733 every 8 connection(s)
I20260927 00:48:55.882898 23170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:48:55.885576 23239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:55.889106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu as pid 23170
I20260927 00:48:55.889261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjP59Ka/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469854491772-392-0/minicluster-data/master-2/wal/instance
I20260927 00:48:55.889154 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:55.889600 392 external_mini_cluster.cc:1428] Setting key d5a0f7473420574c35f1c4e7bcb77b6a
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:48:55.899425 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 3
I20260927 00:48:55.902375 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.891225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.0.98.62:45023)
0927 00:48:55.891385 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.891390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.891758 (+ 368us) server_negotiation.cc:407] Connection header received
0927 00:48:55.892545 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.892548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.892604 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.892688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.893632 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.894199 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.895307 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.895536 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.895612 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.898379 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.898386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.898391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.898417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.900120 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.900670 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.900672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.900673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.900737 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.901111 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.901114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.901116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.901308 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.901427 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.902109 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.902214 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:48:55.903060 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } attempt: 3
I20260927 00:48:55.903337 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.891091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.0.1:57936)
0927 00:48:55.891626 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.891665 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.892339 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.892775 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.892781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.893189 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.893493 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.893505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.894309 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.894312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.895135 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.895143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.895696 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.895701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.895881 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.896636 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.896659 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.898234 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.900240 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.900250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.900262 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.900573 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.900846 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.900849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.900851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.901003 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.901441 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.901448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.901764 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.902050 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.902359 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":5,"spinlock_wait_cycles":47872,"thread_start_us":197,"threads_started":1}
I20260927 00:48:55.903615 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.891092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.60:41733)
0927 00:48:55.891515 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.891531 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.891571 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:48:55.892324 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.892342 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.892542 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.892721 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.893427 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.894501 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.895333 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.895914 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.896007 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.898900 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.898908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.898923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.898951 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.900933 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.901805 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.901810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.901813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.901896 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.902201 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.902204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.902206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.902483 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.902676 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.903301 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.903443 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260927 00:48:55.904970 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:55.914163 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.903943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.0.98.60:41733)
0927 00:48:55.904194 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.904197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.904209 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:48:55.904240 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.904243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.904300 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.904366 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.904873 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.905942 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.906945 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.907171 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.907290 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.909993 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.909997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.909999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.910024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.912127 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.912691 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.912695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.912696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.912787 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.913099 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.913103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.913105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.913314 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.913412 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.913873 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.913991 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":192}
I20260927 00:48:55.914541 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.903476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50664)
0927 00:48:55.903595 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.903609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.903696 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.904472 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.904475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.904650 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.904746 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.904752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.906063 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.906070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.906817 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.906831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.907283 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.907287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.907369 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.908465 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.908478 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.909845 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.912265 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.912270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.912273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.912579 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.912901 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.912904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.912905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.912999 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.913459 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.913462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.913559 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.913664 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.914353 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 00:48:55.914987 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.899858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50662)
0927 00:48:55.900235 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.900250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.900338 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.901053 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.901059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.901212 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.901290 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.901295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.902215 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.902218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.902981 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.902992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.903983 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.903988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.904069 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.904557 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.904570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.905998 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.912195 (+ 6197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.912201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.912206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.912462 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.913059 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.913065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.913069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.913412 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.914192 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.914195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.914309 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.914389 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.914788 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260927 00:48:55.915676 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.899941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.98.60:41733)
0927 00:48:55.900832 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.900844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.900859 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:48:55.900895 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.900898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.900938 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.901019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.901400 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.902086 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.903132 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.903843 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.904281 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.906136 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.906140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.906142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.906164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.912039 (+ 5875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.912836 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.912842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.912846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.912942 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.913548 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.913554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.913557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.913821 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.915284 (+ 1463us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.915383 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.915491 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":113,"threads_started":1}
I20260927 00:48:55.916255 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.905419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56072)
0927 00:48:55.905945 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.905958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.906027 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.906985 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.906989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.907158 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.907271 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.907279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.908179 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.908181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.908691 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.908697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.909559 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.909563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.909677 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.910167 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.910180 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.912164 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.914196 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.914198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.914435 (+ 237us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.914819 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.915116 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.915119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.915121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.915206 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.915788 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.915791 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.915887 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.915995 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.916118 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465}
I20260927 00:48:55.916287 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.906614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.98.61:33177)
0927 00:48:55.906709 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.906713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.906726 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:48:55.906762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.906765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.906808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.906882 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.907403 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.908073 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.908816 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.909431 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.909535 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.912318 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.912322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.912324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.912351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.914083 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.914920 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.914923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.914925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.915007 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.915315 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.915318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.915320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.915600 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.915694 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.916050 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.916159 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 00:48:55.917716 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: Bootstrap starting.
I20260927 00:48:55.918501 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: Bootstrap starting.
I20260927 00:48:55.919910 23163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:55.920324 23088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:55.920583 23163 log.cc:824] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:55.920900 23088 log.cc:824] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:55.921545 23239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:48:55.922717 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: No bootstrap required, opened a new log
I20260927 00:48:55.923273 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: No bootstrap required, opened a new log
I20260927 00:48:55.925433 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.907389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.0.98.62:45023)
0927 00:48:55.907526 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.907532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.907575 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:48:55.907709 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.907714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.907758 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.907853 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.908414 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.909156 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.910347 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.910710 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.910819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.913669 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.913673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.913675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.913696 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.921561 (+ 7865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.922611 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.922614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.922616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.922699 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.923911 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.923915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.923916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.924118 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.924289 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.925169 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.925295 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:48:55.927333 23088 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.927345 23163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.927564 23163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fcfbf7fa134bc393678c04fc3af63c, State: Initialized, Role: FOLLOWER
I20260927 00:48:55.927563 23088 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09b238b83f64b0ea056e40ce3e2df2d, State: Initialized, Role: FOLLOWER
I20260927 00:48:55.927914 23088 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.928175 23163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.929287 23088 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:55.930301 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.930438 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:55.930117 23163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:55.931651 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.922004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41733 (local address 127.0.0.1:50668)
0927 00:48:55.922162 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:55.922176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:55.922269 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:55.922475 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:55.922478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:55.922637 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:55.922722 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.922731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.923687 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.923690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.924477 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:55.924489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.925045 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.925049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:55.925131 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.925742 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:55.925756 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:55.927269 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:55.929666 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.929669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.929671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.929939 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.930239 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:55.930242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:55.930244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:55.930347 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:55.931128 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:55.931131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:55.931206 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:48:55.931326 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:48:55.931473 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 00:48:55.932430 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.922050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.98.60:41733)
0927 00:48:55.922156 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.922160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.922230 (+ 70us) server_negotiation.cc:407] Connection header received
0927 00:48:55.922344 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.922347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.922388 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.922459 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.922825 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.923580 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.924583 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.924928 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.925029 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.927405 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.927408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.927411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.927434 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.929525 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.930060 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.930063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.930064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.930142 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.930784 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.930787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.930789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.930999 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.931094 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.932128 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.932266 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 00:48:55 dist-test-slave-l5sb krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 00:48:55.934556 23260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:55.934625 23260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:55.928802 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.934857 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:55.935773 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: Bootstrap starting.
I20260927 00:48:55.937983 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.927026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.0.98.61:33177)
0927 00:48:55.927119 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.927123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.927269 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:48:55.927462 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.927465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.927508 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.927558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.928436 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.929100 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.930258 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.930596 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.930673 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.933899 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.933903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.933905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.933931 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.936014 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.936495 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.936498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.936500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.936603 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.936904 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.936907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.936909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.937109 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.937293 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.937710 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.937849 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:48:55.939522 23239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: Neither blocks nor log segments found. Creating new log.
W20260927 00:48:55.940313 23270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:55.940421 23270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:55.940402 23239 log.cc:824] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:55.939514 23095 master_runner.cc:186] Error getting master registration for 127.0.98.60:41733: OK, instance_id { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" instance_seqno: 1790470135879877 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:48:55.943920 23095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:48:55.942329 23020 master_runner.cc:186] Error getting master registration for 127.0.98.60:41733: OK, instance_id { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" instance_seqno: 1790470135879877 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:48:55.945676 23020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:48:55.946489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:48:55.947621 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: No bootstrap required, opened a new log
I20260927 00:48:55.947707 23062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b09b238b83f64b0ea056e40ce3e2df2d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952
I20260927 00:48:55.947984 23062 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:48:55.948102 23062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:55.949426 23062 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.950418 23062 leader_election.cc:288] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [CANDIDATE]: Term 1 election: Requested vote from peers 60fcfbf7fa134bc393678c04fc3af63c (127.0.98.61:33177), f1fc9ba6b9fe47988d38c907ae7b48bf (127.0.98.60:41733)
I20260927 00:48:55.951171 23137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60fcfbf7fa134bc393678c04fc3af63c"
I20260927 00:48:55.951442 23137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:55.952148 23239 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.952420 23239 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fc9ba6b9fe47988d38c907ae7b48bf, State: Initialized, Role: FOLLOWER
I20260927 00:48:55.952737 23137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09b238b83f64b0ea056e40ce3e2df2d in term 1.
I20260927 00:48:55.952764 23239 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.953837 23036 leader_election.cc:302] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60fcfbf7fa134bc393678c04fc3af63c, b09b238b83f64b0ea056e40ce3e2df2d; no voters:
I20260927 00:48:55.954138 23247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:55.954363 23247 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [term 1 LEADER]: Becoming Leader. State: Replica: b09b238b83f64b0ea056e40ce3e2df2d, State: Running, Role: LEADER
I20260927 00:48:55.954382 23239 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:55.954684 23247 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } }
I20260927 00:48:55.956440 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09b238b83f64b0ea056e40ce3e2df2d. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.956561 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:55.956775 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf"
I20260927 00:48:55.957270 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:55.958607 23213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09b238b83f64b0ea056e40ce3e2df2d in term 1.
I20260927 00:48:55.961928 23288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:55.963689 23288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:55.970250 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.970510 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:48:55.971249 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:48:55.971735 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:48:55.979518 23137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [term 1 FOLLOWER]: Refusing update from remote peer b09b238b83f64b0ea056e40ce3e2df2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:55.980742 23213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [term 1 FOLLOWER]: Refusing update from remote peer b09b238b83f64b0ea056e40ce3e2df2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:55.980906 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:55.981274 23247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:55.985536 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.966748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.60:41733)
0927 00:48:55.966850 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.966854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.971581 (+ 4727us) server_negotiation.cc:407] Connection header received
0927 00:48:55.971720 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.971723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.971768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.971840 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.975203 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.975985 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.977101 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.977443 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.977512 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.980081 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.980085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.980088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.980115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.982998 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.984234 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.984238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.984240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.984338 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.984623 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.984626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.984628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.984836 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.984942 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.985257 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.985382 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:48:55.987134 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.961511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.0.98.61:33177)
0927 00:48:55.961632 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.961637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.967575 (+ 5938us) server_negotiation.cc:407] Connection header received
0927 00:48:55.967755 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.967759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.967809 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.967861 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.969146 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.969963 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.973427 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.973801 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.973901 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.977396 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.977400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.977403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.977430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.982679 (+ 5249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.983190 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.983194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.983196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.983278 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.986180 (+ 2902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.986184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.986186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.986426 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.986524 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.986869 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.986988 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:48:55.988314 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09b238b83f64b0ea056e40ce3e2df2d. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.988432 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:55.991041 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:55.966117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.0.98.62:45023)
0927 00:48:55.966226 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:55.966231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:55.969077 (+ 2846us) server_negotiation.cc:407] Connection header received
0927 00:48:55.971267 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:55.971271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:55.971319 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:55.971366 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:55.972554 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.973327 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.976824 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:55.977161 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:55.977229 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:55.985407 (+ 8178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:55.985411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:55.985414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:55.985443 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:55.989244 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.989726 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.989730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.989732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.989844 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:55.990129 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:55.990132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:55.990134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:55.990351 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:55.990455 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:55.990777 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 00:48:55.990891 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:48:55.991173 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09b238b83f64b0ea056e40ce3e2df2d. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.991305 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:55.994370 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.994490 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:55.995067 23296 mvcc.cc:205] Tried to move back new op lower bound from 7333765676950994944 to 7333765676873670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333765676950994944}]
I20260927 00:48:55.995602 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:55.995733 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:56.002447 23298 mvcc.cc:205] Tried to move back new op lower bound from 7333765676950994944 to 7333765676873670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333765676950994944}]
I20260927 00:48:56.003741 23288 catalog_manager.cc:1347] Generated new cluster ID: a24af37d63cb4a759a0494607cd13691
I20260927 00:48:56.003810 23288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:56.012807 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:56.012941 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:56.014180 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09b238b83f64b0ea056e40ce3e2df2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } peers { permanent_uuid: "60fcfbf7fa134bc393678c04fc3af63c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "f1fc9ba6b9fe47988d38c907ae7b48bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41733 } } }
I20260927 00:48:56.014308 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:48:56.014950 23300 mvcc.cc:205] Tried to move back new op lower bound from 7333765676950994944 to 7333765676873670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333765676950994944}]
I20260927 00:48:56.040328 23288 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:56.041708 23288 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:56.056504 23288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09b238b83f64b0ea056e40ce3e2df2d: Generated new TSK 0
I20260927 00:48:56.057430 23288 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:48:56.088609 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.085451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.0.98.62:45023)
0927 00:48:56.085582 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.085586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.087490 (+ 1904us) server_negotiation.cc:407] Connection header received
0927 00:48:56.087702 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.087706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.087759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.087816 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.088474 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57966: BlockingRecv error: recv got EOF from 127.0.0.1:57966 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:48:56.090045 23244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.086202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.0.98.60:41733)
0927 00:48:56.086332 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.086336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.089108 (+ 2772us) server_negotiation.cc:407] Connection header received
0927 00:48:56.089297 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.089300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.089350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.089401 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.089924 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50688: BlockingRecv error: recv got EOF from 127.0.0.1:50688 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:48:56.091558 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.086882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.0.98.61:33177)
0927 00:48:56.086995 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.086999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.090413 (+ 3414us) server_negotiation.cc:407] Connection header received
0927 00:48:56.090784 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.090788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.090844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.090899 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.091453 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56110: BlockingRecv error: recv got EOF from 127.0.0.1:56110 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 00:48:56.096442 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.094600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.0.98.61:33177)
0927 00:48:56.094730 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.094733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.095273 (+ 540us) server_negotiation.cc:407] Connection header received
0927 00:48:56.095546 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.095549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.095625 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.095702 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.096331 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56120: BlockingRecv error: recv got EOF from 127.0.0.1:56120 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:48:56.101668 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.100214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.98.61:33177)
0927 00:48:56.100329 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.100333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.100793 (+ 460us) server_negotiation.cc:407] Connection header received
0927 00:48:56.100972 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.100975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.101025 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.101077 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.101570 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56128: BlockingRecv error: recv got EOF from 127.0.0.1:56128 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:48:56.107393 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.105945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.0.98.61:33177)
0927 00:48:56.106039 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.106042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.106508 (+ 466us) server_negotiation.cc:407] Connection header received
0927 00:48:56.106688 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.106691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.106738 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.106788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.107293 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56136: BlockingRecv error: recv got EOF from 127.0.0.1:56136 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:48:56.115106 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.113566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.0.98.61:33177)
0927 00:48:56.113695 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.113699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.114183 (+ 484us) server_negotiation.cc:407] Connection header received
0927 00:48:56.114388 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.114391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.114444 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.114494 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.115014 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56144: BlockingRecv error: recv got EOF from 127.0.0.1:56144 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 00:48:56.127127 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.125425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.0.98.61:33177)
0927 00:48:56.125530 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.125534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.126185 (+ 651us) server_negotiation.cc:407] Connection header received
0927 00:48:56.126393 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.126396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.126445 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.126496 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.127020 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56152: BlockingRecv error: recv got EOF from 127.0.0.1:56152 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 00:48:56.147377 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.145698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.0.98.61:33177)
0927 00:48:56.145859 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.145864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.146386 (+ 522us) server_negotiation.cc:407] Connection header received
0927 00:48:56.146578 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.146581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.146665 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.146718 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.147260 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56164: BlockingRecv error: recv got EOF from 127.0.0.1:56164 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:48:56.183306 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.181951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.61:33177)
0927 00:48:56.182105 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.182111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.182293 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:48:56.182493 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.182496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.182550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.182631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.183173 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56172: BlockingRecv error: recv got EOF from 127.0.0.1:56172 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:48:56.250836 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.249396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.98.61:33177)
0927 00:48:56.249563 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.249567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.249730 (+ 163us) server_negotiation.cc:407] Connection header received
0927 00:48:56.249959 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.249962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.250016 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.250097 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.250608 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56182: BlockingRecv error: recv got EOF from 127.0.0.1:56182 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 00:48:56.381891 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.380667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.0.98.61:33177)
0927 00:48:56.380836 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.380843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.380975 (+ 132us) server_negotiation.cc:407] Connection header received
0927 00:48:56.381095 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.381098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.381148 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.381225 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.381641 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56198: BlockingRecv error: recv got EOF from 127.0.0.1:56198 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 00:48:56.641048 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:56.639863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.98.61:33177)
0927 00:48:56.640014 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:56.640018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:56.640123 (+ 105us) server_negotiation.cc:407] Connection header received
0927 00:48:56.640294 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:56.640298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:56.640353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:56.640444 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:56.640906 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56212: BlockingRecv error: recv got EOF from 127.0.0.1:56212 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:48:56.941634 23270 catalog_manager.cc:1259] Loaded cluster ID: a24af37d63cb4a759a0494607cd13691
I20260927 00:48:56.941722 23270 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: loading cluster ID for follower catalog manager: success
I20260927 00:48:56.943430 23270 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: acquiring CA information for follower catalog manager: success
I20260927 00:48:56.944600 23270 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60fcfbf7fa134bc393678c04fc3af63c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:48:56.973104 23287 catalog_manager.cc:1259] Loaded cluster ID: a24af37d63cb4a759a0494607cd13691
I20260927 00:48:56.973186 23287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: loading cluster ID for follower catalog manager: success
I20260927 00:48:56.974905 23287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: acquiring CA information for follower catalog manager: success
I20260927 00:48:56.976033 23287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1fc9ba6b9fe47988d38c907ae7b48bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:48:57.159605 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:57.155409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.98.61:33177)
0927 00:48:57.155778 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:57.155783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:57.155799 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:48:57.155959 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:57.155963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:57.156109 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:57.156239 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:57.156505 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.157096 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.158067 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.158308 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.158422 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:57.158756 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:57.159199 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:57.159344 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:57.159381 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 00:48:57.159435 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260927 00:48:57.166483 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:57.162887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.0.98.60:41733)
0927 00:48:57.163104 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:57.163108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:57.163122 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:48:57.163290 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:57.163292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:57.163406 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:57.163503 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:57.163754 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.164264 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.165092 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.165309 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.165386 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:57.165644 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:57.166090 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:57.166179 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:57.166308 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:48:57.166359 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260927 00:48:57.173842 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:57.169513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.98.62:45023)
0927 00:48:57.169919 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:57.169924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:57.169941 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:48:57.170036 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:57.170039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:57.170218 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:57.170372 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:57.170576 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.171375 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.172315 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.172659 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.172779 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:57.173101 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:57.173379 (+ 278us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:57.173487 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:57.173565 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 00:48:57.173625 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":71,"threads_started":1}
I20260927 00:48:57.174091 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:57.169954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.0.98.60:41733)
0927 00:48:57.170069 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:57.170073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:57.170220 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:48:57.170397 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:57.170400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:57.170445 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:57.170489 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:57.170856 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.171366 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.172656 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.172989 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.173071 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:57.173581 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:57.173654 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:57.173699 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:57.173919 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:48:57.173986 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:48:57.174118 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:57.169539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.98.61:33177)
0927 00:48:57.169745 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:57.169750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:57.170117 (+ 367us) server_negotiation.cc:407] Connection header received
0927 00:48:57.170234 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:57.170239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:57.170335 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:57.170406 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:57.170809 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.171559 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.172656 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:57.173025 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:57.173092 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:57.173582 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:57.173654 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:57.173699 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:57.173932 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 00:48:57.173991 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 00:48:57.177409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 23020
I20260927 00:48:57.183859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 23095
I20260927 00:48:57.190088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjP59Ka/build/debug/bin/kudu with pid 23170
2026-09-27T00:48:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3108 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6246 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282692 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST